Jul 31 20:30:19.514083 ip-10-0-141-177 systemd[1]: Starting Kubernetes Kubelet... Jul 31 20:30:19.907694 ip-10-0-141-177 kubenswrapper[2544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 20:30:19.907694 ip-10-0-141-177 kubenswrapper[2544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 20:30:19.907694 ip-10-0-141-177 kubenswrapper[2544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 20:30:19.907694 ip-10-0-141-177 kubenswrapper[2544]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 20:30:19.907694 ip-10-0-141-177 kubenswrapper[2544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 20:30:19.910144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.910054 2544 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 20:30:19.914043 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914026 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.914043 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914042 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914047 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914051 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914054 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914057 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914060 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914063 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914066 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914069 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914071 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914074 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914082 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914085 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914088 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914093 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914097 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914100 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914103 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914106 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.914107 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914109 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914111 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914114 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914117 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914120 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914123 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914126 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914128 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914131 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914133 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914136 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914139 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914141 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914144 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914147 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914150 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914153 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914155 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914158 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914160 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.914577 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914163 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914166 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914168 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914172 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914175 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914178 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914180 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914183 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914186 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914188 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914191 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914193 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914195 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914198 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914201 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914205 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914209 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914212 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914215 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914220 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.915343 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914223 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914226 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914229 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914231 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914234 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914237 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914240 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914242 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914245 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914247 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914250 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914252 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914255 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914257 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914260 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914263 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914265 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914268 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914271 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914273 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.915868 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914276 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914278 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914281 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914283 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914286 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914289 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914293 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914296 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914299 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914302 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914304 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914307 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914309 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914312 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914314 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914317 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914319 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914322 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.916352 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.914326 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.918481 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918463 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.918481 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918481 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918485 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918489 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918492 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918495 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918497 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918500 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918503 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918506 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918509 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918512 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918515 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918517 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918520 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918522 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918525 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918528 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918531 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918534 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918537 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.918550 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918540 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918542 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918545 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918548 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918551 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918554 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918557 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918560 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918562 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918565 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918568 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918570 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918573 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918575 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918578 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918581 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918584 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918586 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918588 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918591 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.919181 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918594 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918596 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918599 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918602 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918605 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918607 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918610 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918613 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918615 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918618 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918621 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918624 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918627 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918629 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918633 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918636 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918638 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918641 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918643 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918646 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.919682 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918648 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918651 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918654 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918657 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918659 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918662 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918664 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918667 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918670 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918673 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918675 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918678 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918681 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918683 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918686 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918690 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918694 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918698 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918700 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918703 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.920187 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918706 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918708 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918711 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918714 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918717 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918720 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918723 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918726 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918728 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918731 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918733 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918736 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918739 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918741 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918744 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918746 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918750 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.918754 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.920668 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919168 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919173 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919176 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919179 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919182 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919185 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919188 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919190 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919193 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919196 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919199 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919201 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919204 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919206 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919209 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919211 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919215 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919218 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919221 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.921131 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919224 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919226 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919229 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919232 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919235 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919237 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919240 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919242 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919245 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919248 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919250 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919253 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919255 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919258 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919260 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919263 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919265 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919269 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919272 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.921591 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919275 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919278 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919281 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919283 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919287 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919290 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919293 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919295 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919298 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919302 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919305 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919308 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919311 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919314 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919316 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919319 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919321 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919324 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919326 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919329 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.922082 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919332 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919334 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919337 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919339 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919342 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919345 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919347 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919349 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919357 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919360 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919363 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919365 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919368 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919371 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919374 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919376 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919379 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919382 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919385 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919388 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.922565 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919391 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919393 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919397 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919400 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919404 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919406 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919409 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919412 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919414 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919417 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919419 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919422 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919425 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919429 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919432 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919435 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919438 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919441 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919443 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919446 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.923071 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.919448 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919527 2544 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919539 2544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919552 2544 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919558 2544 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919564 2544 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919569 2544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919578 2544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919582 2544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919586 2544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919589 2544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919592 2544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919595 2544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919599 2544 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919602 2544 flags.go:64] FLAG: --cgroup-root="" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919605 2544 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919608 2544 flags.go:64] FLAG: --client-ca-file="" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919612 2544 flags.go:64] FLAG: --cloud-provider="external" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919615 2544 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919619 2544 flags.go:64] FLAG: --cluster-domain="" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919622 2544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919626 2544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919629 2544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919633 2544 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 20:30:19.923550 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919636 2544 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919639 2544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919642 2544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919646 2544 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919649 2544 flags.go:64] FLAG: --contention-profiling="false" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919652 2544 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919655 2544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919659 2544 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919661 2544 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919666 2544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919669 2544 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919672 2544 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919675 2544 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919678 2544 flags.go:64] FLAG: --enable-server="true" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919681 2544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919687 2544 flags.go:64] FLAG: --event-burst="100" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919691 2544 flags.go:64] FLAG: --event-qps="50" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919694 2544 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919697 2544 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919701 2544 flags.go:64] FLAG: --eviction-hard="" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919704 2544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919707 2544 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919711 2544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919714 2544 flags.go:64] FLAG: --eviction-soft="" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919717 2544 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 20:30:19.924189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919720 2544 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919723 2544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919726 2544 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919729 2544 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919732 2544 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919735 2544 flags.go:64] FLAG: --feature-gates="" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919738 2544 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919741 2544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919744 2544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919748 2544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919751 2544 flags.go:64] FLAG: --healthz-port="10248" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919754 2544 flags.go:64] FLAG: --help="false" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919757 2544 flags.go:64] FLAG: --hostname-override="ip-10-0-141-177.ec2.internal" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919760 2544 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919763 2544 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919766 2544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919769 2544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919772 2544 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919775 2544 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919778 2544 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919781 2544 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919784 2544 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919806 2544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919811 2544 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 20:30:19.924826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919814 2544 flags.go:64] FLAG: --kube-reserved="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919817 2544 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919821 2544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919824 2544 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919826 2544 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919829 2544 flags.go:64] FLAG: --lock-file="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919832 2544 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919835 2544 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919838 2544 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919844 2544 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919847 2544 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919850 2544 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919853 2544 flags.go:64] FLAG: --logging-format="text" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919856 2544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919859 2544 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919862 2544 flags.go:64] FLAG: --manifest-url="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919865 2544 flags.go:64] FLAG: --manifest-url-header="" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919869 2544 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919872 2544 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919876 2544 flags.go:64] FLAG: --max-pods="110" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919879 2544 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919882 2544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919885 2544 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919888 2544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919891 2544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 20:30:19.925417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919894 2544 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919897 2544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919904 2544 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919907 2544 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919910 2544 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919915 2544 flags.go:64] FLAG: --pod-cidr="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919920 2544 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919926 2544 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919929 2544 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919932 2544 flags.go:64] FLAG: --pods-per-core="0" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919935 2544 flags.go:64] FLAG: --port="10250" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919938 2544 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919941 2544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d782e1024f2d191e" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919944 2544 flags.go:64] FLAG: --qos-reserved="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919947 2544 flags.go:64] FLAG: --read-only-port="10255" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919950 2544 flags.go:64] FLAG: --register-node="true" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919953 2544 flags.go:64] FLAG: --register-with-taints="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919957 2544 flags.go:64] FLAG: --registry-burst="10" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919960 2544 flags.go:64] FLAG: --registry-qps="5" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919963 2544 flags.go:64] FLAG: --reserved-cpus="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919966 2544 flags.go:64] FLAG: --reserved-memory="" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919969 2544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919972 2544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919975 2544 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919978 2544 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 20:30:19.926069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919981 2544 flags.go:64] FLAG: --runonce="false" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919984 2544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919987 2544 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919990 2544 flags.go:64] FLAG: --seccomp-default="false" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919993 2544 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919996 2544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.919999 2544 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920002 2544 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920005 2544 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920008 2544 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920011 2544 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920013 2544 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920016 2544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920021 2544 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920024 2544 flags.go:64] FLAG: --system-cgroups="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920027 2544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920032 2544 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920035 2544 flags.go:64] FLAG: --tls-cert-file="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920038 2544 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920042 2544 flags.go:64] FLAG: --tls-min-version="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920045 2544 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920048 2544 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920051 2544 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920054 2544 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920057 2544 flags.go:64] FLAG: --v="2" Jul 31 20:30:19.926687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920061 2544 flags.go:64] FLAG: --version="false" Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920065 2544 flags.go:64] FLAG: --vmodule="" Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920069 2544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.920072 2544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920198 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920202 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920205 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920207 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920210 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920213 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920216 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920218 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920221 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920223 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920226 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920228 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920231 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920234 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920236 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920239 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.927360 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920244 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920247 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920249 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920252 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920254 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920257 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920260 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920263 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920265 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920268 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920271 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920273 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920276 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920278 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920282 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920284 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920287 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920290 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920292 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920295 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.927879 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920297 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920300 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920302 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920305 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920307 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920310 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920312 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920315 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920317 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920320 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920322 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920325 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920330 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920332 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920335 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920338 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920340 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920343 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920345 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920349 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.928377 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920353 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920356 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920358 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920361 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920364 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920366 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920369 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920373 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920376 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920378 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920381 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920383 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920386 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920388 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920391 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920394 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920396 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920399 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920401 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920404 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.928885 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920406 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920409 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920411 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920414 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920419 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920422 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920426 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920430 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920434 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920437 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920440 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920442 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920445 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920448 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920451 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920454 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920456 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920459 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920462 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.929367 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920464 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920467 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920470 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.920472 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.921021 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.927575 2544 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.927592 2544 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.928287 2544 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.928295 2544 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928353 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928358 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928362 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928365 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928368 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928371 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.929850 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928375 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928379 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928383 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928388 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928391 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928394 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928397 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928400 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928403 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928405 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928408 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928411 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928414 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928417 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928421 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928424 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928427 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928430 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928433 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.930226 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928436 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928439 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928441 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928444 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928447 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928450 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928453 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928456 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928458 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928461 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928464 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928466 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928469 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928471 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928474 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928476 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928479 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928481 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928484 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928487 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.930686 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928489 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928492 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928495 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928497 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928499 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928502 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928504 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928507 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928509 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928512 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928515 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928517 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928520 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928522 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928525 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928528 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928531 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928534 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928537 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928540 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.931184 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928542 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928545 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928548 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928551 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928553 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928556 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928559 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928562 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928564 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928567 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928569 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928572 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928575 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928578 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928581 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928584 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928587 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928590 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928593 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.931673 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928596 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928599 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928602 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928604 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928607 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928609 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928612 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928616 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928618 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928621 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928624 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928626 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928629 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928632 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928634 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.932146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.928639 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928753 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928757 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928761 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928764 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928767 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928770 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928773 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928776 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928779 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928782 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928797 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928801 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928803 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928808 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928812 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928816 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928820 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928823 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 20:30:19.932529 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928826 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928829 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928832 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928835 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928838 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928841 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928844 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928847 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928849 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928852 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928855 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928858 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928860 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928863 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928865 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928868 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928870 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928873 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928875 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928878 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 20:30:19.933000 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928881 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928883 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928886 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928889 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928891 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928894 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928897 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928900 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928902 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928905 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928907 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928911 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928914 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928918 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928921 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928924 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928926 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928929 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928932 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928935 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 20:30:19.933468 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928938 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928941 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928944 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928946 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928949 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928951 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928954 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928956 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928959 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928961 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928964 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928967 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928970 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928972 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928975 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928977 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928980 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928983 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928985 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928988 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 20:30:19.934047 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928990 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928993 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928996 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.928998 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929000 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929003 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929005 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929008 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929010 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929013 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929015 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929021 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929024 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929027 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929029 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929032 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929034 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929037 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929039 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 20:30:19.934535 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929042 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:19.929045 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.929050 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.929156 2544 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.932320 2544 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.933570 2544 server.go:1019] "Starting client certificate rotation" Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.933672 2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 20:30:19.935009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.933708 2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 20:30:19.935969 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.935949 2544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 20:30:19.937445 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.937421 2544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 20:30:19.947408 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.947386 2544 log.go:25] "Validated CRI v1 runtime API" Jul 31 20:30:19.949451 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.949436 2544 log.go:25] "Validated CRI v1 image API" Jul 31 20:30:19.951084 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.951060 2544 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 20:30:19.953197 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.953176 2544 fs.go:135] Filesystem UUIDs: map[4b48bfaf-1c3a-403f-badf-87108497d236:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f0ee88b5-1ee2-41df-8e66-c82f5df09117:/dev/nvme0n1p3] Jul 31 20:30:19.953269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.953198 2544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 20:30:19.958836 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.958703 2544 manager.go:216] Machine: {Timestamp:2026-07-31 20:30:19.95762828 +0000 UTC m=+0.319079329 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100392 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d94dc59b233e14838215a9d8f698f SystemUUID:ec2d94dc-59b2-33e1-4838-215a9d8f698f BootID:531facbf-e180-4f83-b45d-c5f40703c77d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d0:72:5a:6d:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d0:72:5a:6d:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:f2:33:2a:51:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 20:30:19.958836 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.958829 2544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 20:30:19.958954 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.958910 2544 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 20:30:19.961108 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.961090 2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 20:30:19.961486 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.961460 2544 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 20:30:19.961623 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.961490 2544 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 20:30:19.962107 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.962094 2544 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 20:30:19.962144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.962109 2544 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 20:30:19.962144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.962116 2544 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 20:30:19.962701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.962691 2544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 20:30:19.963384 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.963373 2544 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 20:30:19.966116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966104 2544 state_mem.go:36] "Initialized new in-memory state store" Jul 31 20:30:19.966289 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966279 2544 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 20:30:19.966320 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966310 2544 kubelet.go:486] "Attempting to sync node with API server" Jul 31 20:30:19.966347 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966321 2544 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 20:30:19.966347 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966331 2544 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 20:30:19.966347 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966341 2544 kubelet.go:398] "Adding apiserver pod source" Jul 31 20:30:19.966462 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966349 2544 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 20:30:19.966721 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966706 2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 20:30:19.966802 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.966734 2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 20:30:19.969737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.969719 2544 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 20:30:19.971321 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.971303 2544 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 20:30:19.971403 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.971336 2544 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 20:30:19.972194 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972183 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972199 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972206 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972212 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972220 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972226 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972234 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972239 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 20:30:19.972245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972247 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 20:30:19.972474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972253 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 20:30:19.972474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972262 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 20:30:19.972474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972267 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 20:30:19.972636 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.972612 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 20:30:19.972672 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.972657 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 20:30:19.972869 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972858 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 20:30:19.972869 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.972868 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 20:30:19.977005 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.976940 2544 server.go:1268] "Started kubelet" Jul 31 20:30:19.977130 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.977059 2544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 20:30:19.977130 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.977091 2544 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 20:30:19.977416 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.977393 2544 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 20:30:19.977993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.977972 2544 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 20:30:19.978136 ip-10-0-141-177 systemd[1]: Started Kubernetes Kubelet. Jul 31 20:30:19.979409 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.979011 2544 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 20:30:19.980391 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.980375 2544 server.go:310] "Adding debug handlers to kubelet server" Jul 31 20:30:19.983212 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.983184 2544 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 20:30:19.984839 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.983923 2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-177.ec2.internal.18c779e93f972771 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-177.ec2.internal,UID:ip-10-0-141-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-177.ec2.internal,},FirstTimestamp:2026-07-31 20:30:19.976902513 +0000 UTC m=+0.338353530,LastTimestamp:2026-07-31 20:30:19.976902513 +0000 UTC m=+0.338353530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-177.ec2.internal,}" Jul 31 20:30:19.984926 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.984907 2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4xvrs" Jul 31 20:30:19.985249 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985229 2544 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 20:30:19.985249 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985244 2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 20:30:19.985596 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.985580 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:19.985818 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985782 2544 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 20:30:19.985898 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985821 2544 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 20:30:19.985898 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985823 2544 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 20:30:19.985991 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985926 2544 reconstruct.go:97] "Volume reconstruction finished" Jul 31 20:30:19.985991 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.985934 2544 reconciler.go:29] "Reconciler: start to sync state" Jul 31 20:30:19.986128 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986107 2544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 20:30:19.986128 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986129 2544 factory.go:55] Registering systemd factory Jul 31 20:30:19.986230 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986141 2544 factory.go:223] Registration of the systemd container factory successfully Jul 31 20:30:19.986462 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986450 2544 factory.go:153] Registering CRI-O factory Jul 31 20:30:19.986462 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986463 2544 factory.go:223] Registration of the crio container factory successfully Jul 31 20:30:19.986553 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986482 2544 factory.go:103] Registering Raw factory Jul 31 20:30:19.986553 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986492 2544 manager.go:1196] Started watching for new ooms in manager Jul 31 20:30:19.987005 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.986992 2544 manager.go:349] Starting recovery of all containers Jul 31 20:30:19.991420 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.991396 2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4xvrs" Jul 31 20:30:19.992481 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.992458 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 31 20:30:19.992541 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:19.992492 2544 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 20:30:19.997433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:19.997415 2544 manager.go:354] Recovery completed Jul 31 20:30:20.002891 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.002765 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.004716 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.004701 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.004765 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.004730 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.004765 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.004740 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.005288 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.005274 2544 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 20:30:20.005288 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.005287 2544 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 20:30:20.005369 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.005302 2544 state_mem.go:36] "Initialized new in-memory state store" Jul 31 20:30:20.005369 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.005279 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.006027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006013 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.006102 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006036 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.006102 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006045 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.006102 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.005965 2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-177.ec2.internal.18c779e9413f90c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-177.ec2.internal,UID:ip-10-0-141-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-177.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-177.ec2.internal,},FirstTimestamp:2026-07-31 20:30:20.004716737 +0000 UTC m=+0.366167756,LastTimestamp:2026-07-31 20:30:20.004716737 +0000 UTC m=+0.366167756,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-177.ec2.internal,}" Jul 31 20:30:20.006246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006134 2544 policy_none.go:49] "None policy: Start" Jul 31 20:30:20.006246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006146 2544 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 20:30:20.006246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.006157 2544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 20:30:20.007149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.007138 2544 policy_none.go:47] "Start" Jul 31 20:30:20.037372 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037354 2544 manager.go:337] "Starting Device Plugin manager" Jul 31 20:30:20.037448 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.037386 2544 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 20:30:20.037448 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037409 2544 server.go:87] "Starting device plugin registration server" Jul 31 20:30:20.037690 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037678 2544 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 20:30:20.037726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037694 2544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 20:30:20.037826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037777 2544 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 20:30:20.037908 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037895 2544 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 20:30:20.037962 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.037910 2544 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 20:30:20.038396 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.038375 2544 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 20:30:20.038497 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.038430 2544 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.119613 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.119570 2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 20:30:20.121093 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.121072 2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 20:30:20.121192 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.121109 2544 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 20:30:20.121192 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.121133 2544 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 20:30:20.121192 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.121165 2544 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 20:30:20.123522 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.123498 2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 20:30:20.138647 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.138628 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.139472 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.139448 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.139581 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.139487 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.139581 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.139502 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.139581 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.139523 2544 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.148827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.148808 2544 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.148891 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.148830 2544 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-177.ec2.internal\": node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.162269 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.162211 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.221772 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.221745 2544 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal"] Jul 31 20:30:20.221924 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.221842 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.223216 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.223197 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.223332 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.223234 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.223332 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.223249 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.224368 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.224356 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.224478 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.224461 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.224560 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.224498 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.225075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225058 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.225161 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225091 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.225161 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225105 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.225259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225103 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.225259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225192 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.225259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.225204 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.226097 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.226080 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.226179 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.226110 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 20:30:20.226748 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.226734 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientMemory" Jul 31 20:30:20.226836 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.226762 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 20:30:20.226836 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.226775 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeHasSufficientPID" Jul 31 20:30:20.247394 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.247371 2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-177.ec2.internal\" not found" node="ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.251983 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.251967 2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-177.ec2.internal\" not found" node="ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.262567 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.262549 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.363257 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.363223 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.387563 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.387535 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.387684 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.387566 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.387684 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.387589 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ceb4d3aec329af764e1753333df00b0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-177.ec2.internal\" (UID: \"9ceb4d3aec329af764e1753333df00b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.463896 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.463807 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.488310 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488280 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.488397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488314 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.488397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488336 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ceb4d3aec329af764e1753333df00b0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-177.ec2.internal\" (UID: \"9ceb4d3aec329af764e1753333df00b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.488397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488387 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ceb4d3aec329af764e1753333df00b0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-177.ec2.internal\" (UID: \"9ceb4d3aec329af764e1753333df00b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.488397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488390 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.488517 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.488387 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc51ba05675423a991fa65b5ef840fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal\" (UID: \"fc51ba05675423a991fa65b5ef840fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.549397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.549364 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.554637 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:20.554614 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:20.564421 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.564404 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.664530 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.664486 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.765162 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.765085 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:20.865831 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:20.865780 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:22.918376 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:22.918343 2544 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 20:30:22.918807 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:22.918517 2544 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 20:30:22.951707 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:22.951671 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:22.970577 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:22.970552 2544 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 20:30:22.980703 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:22.980673 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 20:25:21 +0000 UTC" deadline="2028-05-16 03:38:36.596585338 +0000 UTC" Jul 31 20:30:22.980703 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:22.980699 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15703h8m13.615889009s" Jul 31 20:30:23.006425 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.006352 2544 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 20:30:23.052389 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:23.052348 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:23.102998 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:23.102967 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc51ba05675423a991fa65b5ef840fb8.slice/crio-7bbabca9cb7ecd2084d997d88332b69185d58b10a208f53edbee6f123c097e3e WatchSource:0}: Error finding container 7bbabca9cb7ecd2084d997d88332b69185d58b10a208f53edbee6f123c097e3e: Status 404 returned error can't find the container with id 7bbabca9cb7ecd2084d997d88332b69185d58b10a208f53edbee6f123c097e3e Jul 31 20:30:23.103208 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:23.103185 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceb4d3aec329af764e1753333df00b0.slice/crio-e157bcb08b3f6140639e540f186583ba57e9cccfebb18ef11df3febcdedb25c2 WatchSource:0}: Error finding container e157bcb08b3f6140639e540f186583ba57e9cccfebb18ef11df3febcdedb25c2: Status 404 returned error can't find the container with id e157bcb08b3f6140639e540f186583ba57e9cccfebb18ef11df3febcdedb25c2 Jul 31 20:30:23.109482 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.109464 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:30:23.110743 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.110696 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" event={"ID":"9ceb4d3aec329af764e1753333df00b0","Type":"ContainerStarted","Data":"e157bcb08b3f6140639e540f186583ba57e9cccfebb18ef11df3febcdedb25c2"} Jul 31 20:30:23.111858 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.111830 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" event={"ID":"fc51ba05675423a991fa65b5ef840fb8","Type":"ContainerStarted","Data":"7bbabca9cb7ecd2084d997d88332b69185d58b10a208f53edbee6f123c097e3e"} Jul 31 20:30:23.153375 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:23.153330 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:23.253885 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:23.253844 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-177.ec2.internal\" not found" Jul 31 20:30:23.325807 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.325738 2544 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 20:30:23.370819 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.370788 2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" Jul 31 20:30:23.379721 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.379695 2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 20:30:23.380640 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.380627 2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" Jul 31 20:30:23.390526 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.390507 2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 20:30:23.958360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.958076 2544 apiserver.go:52] "Watching apiserver" Jul 31 20:30:23.968369 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.968150 2544 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 20:30:23.969597 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.969140 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-g8cfg","openshift-multus/network-metrics-daemon-285jk","openshift-network-operator/iptables-alerter-bxvp5","openshift-ovn-kubernetes/ovnkube-node-676fq","kube-system/global-pull-secret-syncer-b6gsd","kube-system/konnectivity-agent-x89qb","kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc","openshift-cluster-node-tuning-operator/tuned-t94kz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal","openshift-multus/multus-2tm82","openshift-network-diagnostics/network-check-target-nm599","openshift-dns/node-resolver-v8hvl","openshift-image-registry/node-ca-68whh"] Jul 31 20:30:23.973077 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.971576 2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 20:30:23.973077 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.972415 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:23.975408 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.975381 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:23.975539 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:23.975459 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:23.975608 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.975538 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6zphn\"" Jul 31 20:30:23.975684 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.975665 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.975743 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.975711 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.977789 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.977769 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:23.979988 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.979966 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:23.980701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.980566 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.980701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.980592 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.980701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.980595 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ht5s9\"" Jul 31 20:30:23.980701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.980689 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 20:30:23.982119 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982079 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-hosts-file\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:23.982119 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982112 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-tmp-dir\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:23.982259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982139 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d77\" (UniqueName: \"kubernetes.io/projected/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-kube-api-access-v5d77\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:23.982441 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982356 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.983166 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982781 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 20:30:23.983166 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982799 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.983166 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.982854 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8qhn9\"" Jul 31 20:30:23.983896 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.983603 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 20:30:23.983896 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.983761 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 20:30:23.986955 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.985965 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:23.986955 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:23.986059 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:23.986955 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.986888 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 20:30:23.988520 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.988498 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:23.988708 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.988689 2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 20:30:23.990774 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.990757 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 20:30:23.991222 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.991202 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 20:30:23.991320 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.991299 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-48qf2\"" Jul 31 20:30:23.991600 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.991580 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:23.993571 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.993550 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:23.994103 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994086 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 20:30:23.994370 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994346 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 20:30:23.994488 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994380 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.994551 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994108 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 20:30:23.994551 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994513 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.994665 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.994593 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5t7pr\"" Jul 31 20:30:23.995717 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.995699 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 20:30:23.996621 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.996064 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:23.996621 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.996132 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.996621 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.996285 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.996621 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.996070 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sklbh\"" Jul 31 20:30:23.998714 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.998695 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 20:30:23.999063 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.999042 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w9mfk\"" Jul 31 20:30:23.999240 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.999226 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 20:30:23.999436 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:23.999419 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.000956 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.000937 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.002461 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.001733 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 20:30:24.002461 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.002019 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 20:30:24.002461 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.002251 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlqkm\"" Jul 31 20:30:24.002645 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.002468 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 20:30:24.003147 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.003130 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 20:30:24.003418 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.003399 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w8rws\"" Jul 31 20:30:24.005311 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.005290 2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dbdrj" Jul 31 20:30:24.005311 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.005293 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:24.005452 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.005378 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:24.012734 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.012612 2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dbdrj" Jul 31 20:30:24.072353 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.072323 2544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083157 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-conf-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083200 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-multus-certs\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083227 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnjkk\" (UniqueName: \"kubernetes.io/projected/2dda82ab-5555-4f13-b159-df10a22d5b36-kube-api-access-mnjkk\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083253 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083282 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovn-node-metrics-cert\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083305 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-script-lib\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083326 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-kubelet-config\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083349 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083374 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-tuned\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083398 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083420 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-hostroot\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083451 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083477 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysconfig\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083491 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-systemd\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083513 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-lib-modules\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083532 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-var-lib-kubelet\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083549 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083573 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-env-overrides\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083597 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/fafcf7c3-be67-4f67-a14b-6147b5845d1e-kube-api-access-cjzn6\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083620 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-os-release\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083650 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083675 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprxb\" (UniqueName: \"kubernetes.io/projected/f92dc492-6985-4129-817c-60cbd61a0d6e-kube-api-access-jprxb\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083697 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-conf\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083727 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlkff\" (UniqueName: \"kubernetes.io/projected/585a0b01-8e0b-4f08-a98e-33508e42d0bd-kube-api-access-mlkff\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083754 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-kubelet\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083778 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-ovn\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083799 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-log-socket\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083828 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083843 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-os-release\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083858 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-netns\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083888 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-systemd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.084907 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083911 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-etc-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.083935 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-config\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084063 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9027ab96-d2cc-474c-b92d-9865ba493785-konnectivity-ca\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084127 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084179 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-etc-kubernetes\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084217 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-kubernetes\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084254 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d77\" (UniqueName: \"kubernetes.io/projected/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-kube-api-access-v5d77\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084279 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-netns\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084316 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-bin\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084399 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-netd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084431 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwlp\" (UniqueName: \"kubernetes.io/projected/c6a651d5-2e0e-490d-95b5-15f7165eba9f-kube-api-access-tjwlp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084569 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzs72\" (UniqueName: \"kubernetes.io/projected/eb64b08a-e299-4eba-9311-0126eb3f0792-kube-api-access-tzs72\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084606 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9027ab96-d2cc-474c-b92d-9865ba493785-agent-certs\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084634 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084661 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-k8s-cni-cncf-io\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084699 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-daemon-config\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.085585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084733 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084764 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-kubelet\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084789 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-systemd-units\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084810 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-run\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084827 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-socket-dir-parent\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084850 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-multus\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084898 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-node-log\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084925 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-dbus\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084948 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.084973 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/585a0b01-8e0b-4f08-a98e-33508e42d0bd-serviceca\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085004 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-cni-binary-copy\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085028 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-bin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085071 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-system-cni-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085101 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb64b08a-e299-4eba-9311-0126eb3f0792-host-slash\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085138 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-hosts-file\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085166 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-system-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085191 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.086186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085217 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-sys-fs\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085251 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-cnibin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085277 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-tmp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085307 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-hosts-file\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085312 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgpl\" (UniqueName: \"kubernetes.io/projected/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-kube-api-access-qxgpl\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085356 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb64b08a-e299-4eba-9311-0126eb3f0792-iptables-alerter-script\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085393 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-device-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085428 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085456 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-cnibin\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085489 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-socket-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085516 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-registration-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085541 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-var-lib-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085560 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085608 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-sys\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085626 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-host\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085642 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qq5\" (UniqueName: \"kubernetes.io/projected/72af3523-18bb-4a16-a0d4-b6db4881ce46-kube-api-access-d6qq5\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.086930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085658 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-slash\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.087687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085676 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-modprobe-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.087687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085692 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/585a0b01-8e0b-4f08-a98e-33508e42d0bd-host\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.087687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.085712 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-tmp-dir\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.087687 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.086108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-tmp-dir\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.090629 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.090220 2544 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 20:30:24.097763 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.097733 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d77\" (UniqueName: \"kubernetes.io/projected/c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3-kube-api-access-v5d77\") pod \"node-resolver-v8hvl\" (UID: \"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3\") " pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.186831 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186796 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-system-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.186831 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186837 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186886 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-sys-fs\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186938 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-cnibin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186971 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-tmp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186972 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-sys-fs\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.186988 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgpl\" (UniqueName: \"kubernetes.io/projected/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-kube-api-access-qxgpl\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187019 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb64b08a-e299-4eba-9311-0126eb3f0792-iptables-alerter-script\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187028 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187045 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-device-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187081 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187098 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-device-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187109 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-cnibin\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.187116 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-cnibin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187133 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-socket-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187158 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-registration-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187176 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-var-lib-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187198 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187221 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-sys\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187242 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-host\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187261 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qq5\" (UniqueName: \"kubernetes.io/projected/72af3523-18bb-4a16-a0d4-b6db4881ce46-kube-api-access-d6qq5\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187263 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187301 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-slash\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187339 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-modprobe-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187365 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/585a0b01-8e0b-4f08-a98e-33508e42d0bd-host\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187392 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-conf-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187441 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-multus-certs\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187469 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjkk\" (UniqueName: \"kubernetes.io/projected/2dda82ab-5555-4f13-b159-df10a22d5b36-kube-api-access-mnjkk\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187499 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187527 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovn-node-metrics-cert\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.187737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187570 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-script-lib\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187594 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/585a0b01-8e0b-4f08-a98e-33508e42d0bd-host\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187633 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-socket-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187642 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-conf-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187645 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-kubelet-config\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187679 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-system-cni-dir\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187686 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-registration-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187313 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-cnibin\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187696 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-multus-certs\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187728 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-slash\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187769 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb64b08a-e299-4eba-9311-0126eb3f0792-iptables-alerter-script\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187808 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-modprobe-d\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187771 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-var-lib-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187599 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-kubelet-config\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187842 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-sys\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187842 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-host\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187807 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187890 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.188720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187925 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-tuned\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187963 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187991 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-hostroot\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.188012 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188026 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188059 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysconfig\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.188095 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:24.688057054 +0000 UTC m=+3.064430220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188122 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysconfig\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188173 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-systemd\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.187898 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188330 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-hostroot\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.188405 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.188445 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:24.688431987 +0000 UTC m=+3.064805147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188123 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-systemd\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188539 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-lib-modules\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188571 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-var-lib-kubelet\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188599 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.189755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188623 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-env-overrides\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188644 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-script-lib\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188653 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/fafcf7c3-be67-4f67-a14b-6147b5845d1e-kube-api-access-cjzn6\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188682 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-os-release\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188707 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188711 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188740 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprxb\" (UniqueName: \"kubernetes.io/projected/f92dc492-6985-4129-817c-60cbd61a0d6e-kube-api-access-jprxb\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188764 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-var-lib-kubelet\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188768 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-conf\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188809 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlkff\" (UniqueName: \"kubernetes.io/projected/585a0b01-8e0b-4f08-a98e-33508e42d0bd-kube-api-access-mlkff\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188837 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-kubelet\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188858 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-ovn\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188904 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-log-socket\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188931 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188935 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-sysctl-conf\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188950 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188955 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-os-release\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.190720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188977 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-netns\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.188998 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-systemd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189021 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-etc-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189043 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-config\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189067 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9027ab96-d2cc-474c-b92d-9865ba493785-konnectivity-ca\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189089 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189111 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-etc-kubernetes\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189172 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-kubernetes\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189218 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-netns\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189245 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-bin\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189269 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-netd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189271 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-env-overrides\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189280 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-os-release\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189295 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwlp\" (UniqueName: \"kubernetes.io/projected/c6a651d5-2e0e-490d-95b5-15f7165eba9f-kube-api-access-tjwlp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189320 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzs72\" (UniqueName: \"kubernetes.io/projected/eb64b08a-e299-4eba-9311-0126eb3f0792-kube-api-access-tzs72\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189399 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-lib-modules\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189599 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-kubelet\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.191728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189646 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-ovn\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189667 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9027ab96-d2cc-474c-b92d-9865ba493785-agent-certs\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189683 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-log-socket\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189708 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189747 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-k8s-cni-cncf-io\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189775 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-daemon-config\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189784 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189804 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189835 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189835 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-kubelet\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189888 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-systemd-units\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189905 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-kubelet\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189917 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-run\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189949 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-socket-dir-parent\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189960 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-run\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.189983 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-multus\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190012 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-node-log\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.193000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190025 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-systemd-units\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190039 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-dbus\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190067 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190095 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/585a0b01-8e0b-4f08-a98e-33508e42d0bd-serviceca\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190122 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-cni-binary-copy\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190150 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-bin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190160 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f92dc492-6985-4129-817c-60cbd61a0d6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190177 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-system-cni-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190207 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb64b08a-e299-4eba-9311-0126eb3f0792-host-slash\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190260 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72af3523-18bb-4a16-a0d4-b6db4881ce46-etc-selinux\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190295 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb64b08a-e299-4eba-9311-0126eb3f0792-host-slash\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190304 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-etc-kubernetes\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190348 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-k8s-cni-cncf-io\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190358 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-kubernetes\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190426 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-bin\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190467 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-cni-netd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190466 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9027ab96-d2cc-474c-b92d-9865ba493785-konnectivity-ca\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190523 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-os-release\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.194049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190568 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-run-netns\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190606 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-run-systemd\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190644 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-etc-openvswitch\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190764 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99e6c527-9518-4a36-81c2-ea2634d4ecca-dbus\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190968 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-socket-dir-parent\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.190969 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-multus-daemon-config\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191024 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-node-log\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191071 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-bin\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191143 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovnkube-config\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191191 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191217 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f92dc492-6985-4129-817c-60cbd61a0d6e-system-cni-dir\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191259 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fafcf7c3-be67-4f67-a14b-6147b5845d1e-host-run-netns\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191578 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/585a0b01-8e0b-4f08-a98e-33508e42d0bd-serviceca\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191626 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dda82ab-5555-4f13-b159-df10a22d5b36-host-var-lib-cni-multus\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191706 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dda82ab-5555-4f13-b159-df10a22d5b36-cni-binary-copy\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.191763 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-etc-tuned\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.194223 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9027ab96-d2cc-474c-b92d-9865ba493785-agent-certs\") pod \"konnectivity-agent-x89qb\" (UID: \"9027ab96-d2cc-474c-b92d-9865ba493785\") " pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.195404 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6a651d5-2e0e-490d-95b5-15f7165eba9f-tmp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.197010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.196182 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgpl\" (UniqueName: \"kubernetes.io/projected/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-kube-api-access-qxgpl\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.199111 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.198491 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprxb\" (UniqueName: \"kubernetes.io/projected/f92dc492-6985-4129-817c-60cbd61a0d6e-kube-api-access-jprxb\") pod \"multus-additional-cni-plugins-g8cfg\" (UID: \"f92dc492-6985-4129-817c-60cbd61a0d6e\") " pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.199111 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.198724 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fafcf7c3-be67-4f67-a14b-6147b5845d1e-ovn-node-metrics-cert\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.199111 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.198942 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:24.199111 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.199045 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:24.199111 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.199062 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:24.199375 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.199140 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:24.699119618 +0000 UTC m=+3.075492792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:24.199941 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.199915 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qq5\" (UniqueName: \"kubernetes.io/projected/72af3523-18bb-4a16-a0d4-b6db4881ce46-kube-api-access-d6qq5\") pod \"aws-ebs-csi-driver-node-gr6tc\" (UID: \"72af3523-18bb-4a16-a0d4-b6db4881ce46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.202946 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.202158 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwlp\" (UniqueName: \"kubernetes.io/projected/c6a651d5-2e0e-490d-95b5-15f7165eba9f-kube-api-access-tjwlp\") pod \"tuned-t94kz\" (UID: \"c6a651d5-2e0e-490d-95b5-15f7165eba9f\") " pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.203547 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.203525 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjkk\" (UniqueName: \"kubernetes.io/projected/2dda82ab-5555-4f13-b159-df10a22d5b36-kube-api-access-mnjkk\") pod \"multus-2tm82\" (UID: \"2dda82ab-5555-4f13-b159-df10a22d5b36\") " pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.203646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.203610 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzs72\" (UniqueName: \"kubernetes.io/projected/eb64b08a-e299-4eba-9311-0126eb3f0792-kube-api-access-tzs72\") pod \"iptables-alerter-bxvp5\" (UID: \"eb64b08a-e299-4eba-9311-0126eb3f0792\") " pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.203646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.203617 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzn6\" (UniqueName: \"kubernetes.io/projected/fafcf7c3-be67-4f67-a14b-6147b5845d1e-kube-api-access-cjzn6\") pod \"ovnkube-node-676fq\" (UID: \"fafcf7c3-be67-4f67-a14b-6147b5845d1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.205082 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.205063 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlkff\" (UniqueName: \"kubernetes.io/projected/585a0b01-8e0b-4f08-a98e-33508e42d0bd-kube-api-access-mlkff\") pod \"node-ca-68whh\" (UID: \"585a0b01-8e0b-4f08-a98e-33508e42d0bd\") " pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.230041 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.229962 2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 20:30:24.288264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.288221 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8hvl" Jul 31 20:30:24.298984 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.298954 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxvp5" Jul 31 20:30:24.311900 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.311853 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:24.316617 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.316590 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:24.329468 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.329439 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" Jul 31 20:30:24.335344 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.335313 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" Jul 31 20:30:24.343207 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.343164 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t94kz" Jul 31 20:30:24.350888 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.350855 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68whh" Jul 31 20:30:24.361694 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.361669 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2tm82" Jul 31 20:30:24.694781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.694735 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:24.694781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.694787 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:24.695055 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.694902 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:24.695055 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.694920 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:24.695055 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.694986 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:25.694964929 +0000 UTC m=+4.071338099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:24.695055 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.695006 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:25.694996276 +0000 UTC m=+4.071369433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:24.796038 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:24.796002 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:24.796188 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.796137 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:24.796188 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.796150 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:24.796188 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.796158 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:24.796294 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:24.796206 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:25.796194419 +0000 UTC m=+4.172567575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:25.017916 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:25.017833 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 20:25:24 +0000 UTC" deadline="2028-04-22 05:44:14.4448535 +0000 UTC" Jul 31 20:30:25.017916 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:25.017913 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15129h13m49.426944878s" Jul 31 20:30:25.137831 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.137805 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72af3523_18bb_4a16_a0d4_b6db4881ce46.slice/crio-92a75f84fc2c2ba18ef24e5b893fa1a704fe439a38c4013940b34edb1106e04d WatchSource:0}: Error finding container 92a75f84fc2c2ba18ef24e5b893fa1a704fe439a38c4013940b34edb1106e04d: Status 404 returned error can't find the container with id 92a75f84fc2c2ba18ef24e5b893fa1a704fe439a38c4013940b34edb1106e04d Jul 31 20:30:25.138417 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.138388 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a651d5_2e0e_490d_95b5_15f7165eba9f.slice/crio-8b3d3f924304a5bed4332568774361e59e965dda8d5ac55616a1f452f3b089ee WatchSource:0}: Error finding container 8b3d3f924304a5bed4332568774361e59e965dda8d5ac55616a1f452f3b089ee: Status 404 returned error can't find the container with id 8b3d3f924304a5bed4332568774361e59e965dda8d5ac55616a1f452f3b089ee Jul 31 20:30:25.139771 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.139748 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafcf7c3_be67_4f67_a14b_6147b5845d1e.slice/crio-ef68cc3a66e0fdae66af50d7cb5382aac7f288923ad4bbad2022f212547dafb1 WatchSource:0}: Error finding container ef68cc3a66e0fdae66af50d7cb5382aac7f288923ad4bbad2022f212547dafb1: Status 404 returned error can't find the container with id ef68cc3a66e0fdae66af50d7cb5382aac7f288923ad4bbad2022f212547dafb1 Jul 31 20:30:25.151344 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.143947 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ab9526_3efa_4a3c_82aa_b57b0b56d2f3.slice/crio-8296829b33d1b620a736cdcfdec5e73cf02169373a93eb5400c870ccbd03c7ba WatchSource:0}: Error finding container 8296829b33d1b620a736cdcfdec5e73cf02169373a93eb5400c870ccbd03c7ba: Status 404 returned error can't find the container with id 8296829b33d1b620a736cdcfdec5e73cf02169373a93eb5400c870ccbd03c7ba Jul 31 20:30:25.165486 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.165460 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb64b08a_e299_4eba_9311_0126eb3f0792.slice/crio-3c9bef19ed43a1f660144665a49a32bc3d00a1035fa1e7e771823e9d25515014 WatchSource:0}: Error finding container 3c9bef19ed43a1f660144665a49a32bc3d00a1035fa1e7e771823e9d25515014: Status 404 returned error can't find the container with id 3c9bef19ed43a1f660144665a49a32bc3d00a1035fa1e7e771823e9d25515014 Jul 31 20:30:25.166362 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.166337 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92dc492_6985_4129_817c_60cbd61a0d6e.slice/crio-71571f8663bd1f75064c213e018309e57a630133178d57d6dab3a73a6d756f3d WatchSource:0}: Error finding container 71571f8663bd1f75064c213e018309e57a630133178d57d6dab3a73a6d756f3d: Status 404 returned error can't find the container with id 71571f8663bd1f75064c213e018309e57a630133178d57d6dab3a73a6d756f3d Jul 31 20:30:25.167214 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.167061 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585a0b01_8e0b_4f08_a98e_33508e42d0bd.slice/crio-5fd39e1e25bf0f90045059e688308d40c0cd59688f4acbd93f9d850119b2f4df WatchSource:0}: Error finding container 5fd39e1e25bf0f90045059e688308d40c0cd59688f4acbd93f9d850119b2f4df: Status 404 returned error can't find the container with id 5fd39e1e25bf0f90045059e688308d40c0cd59688f4acbd93f9d850119b2f4df Jul 31 20:30:25.168315 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:30:25.168295 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dda82ab_5555_4f13_b159_df10a22d5b36.slice/crio-a7ecf02320831938785c24d7b01a2430115aa32e14dffe90d2f8f4ff933aced1 WatchSource:0}: Error finding container a7ecf02320831938785c24d7b01a2430115aa32e14dffe90d2f8f4ff933aced1: Status 404 returned error can't find the container with id a7ecf02320831938785c24d7b01a2430115aa32e14dffe90d2f8f4ff933aced1 Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:25.702948 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.703243 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.703318 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:27.703298889 +0000 UTC m=+6.079672053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.703330 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:25.703244 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:25.703492 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.703381 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:27.703364709 +0000 UTC m=+6.079737869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:25.804726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:25.804691 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:25.804923 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.804849 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:25.804923 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.804889 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:25.804923 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.804901 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:25.805097 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:25.804958 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:27.804939519 +0000 UTC m=+6.181312677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:26.019010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.018881 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 20:25:24 +0000 UTC" deadline="2028-02-29 06:48:35.090976772 +0000 UTC" Jul 31 20:30:26.019010 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.018960 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13858h18m9.07202351s" Jul 31 20:30:26.109102 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.109040 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:26.109271 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:26.109190 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:26.109587 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.109566 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:26.109699 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:26.109677 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:26.109761 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.109698 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:26.109817 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:26.109784 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:26.119751 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.119717 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" event={"ID":"72af3523-18bb-4a16-a0d4-b6db4881ce46","Type":"ContainerStarted","Data":"92a75f84fc2c2ba18ef24e5b893fa1a704fe439a38c4013940b34edb1106e04d"} Jul 31 20:30:26.129386 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.129336 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68whh" event={"ID":"585a0b01-8e0b-4f08-a98e-33508e42d0bd","Type":"ContainerStarted","Data":"5fd39e1e25bf0f90045059e688308d40c0cd59688f4acbd93f9d850119b2f4df"} Jul 31 20:30:26.141988 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.141949 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerStarted","Data":"71571f8663bd1f75064c213e018309e57a630133178d57d6dab3a73a6d756f3d"} Jul 31 20:30:26.153078 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.153041 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxvp5" event={"ID":"eb64b08a-e299-4eba-9311-0126eb3f0792","Type":"ContainerStarted","Data":"3c9bef19ed43a1f660144665a49a32bc3d00a1035fa1e7e771823e9d25515014"} Jul 31 20:30:26.155728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.155695 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"ef68cc3a66e0fdae66af50d7cb5382aac7f288923ad4bbad2022f212547dafb1"} Jul 31 20:30:26.163800 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.163767 2544 generic.go:356] "Generic (PLEG): container finished" podID="fc51ba05675423a991fa65b5ef840fb8" containerID="8a3a67e6aefaaeeab3d9eb8ba058fd2f4eb8a17f68caae0a133c8b49d3c9bd1b" exitCode=0 Jul 31 20:30:26.163976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.163859 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" event={"ID":"fc51ba05675423a991fa65b5ef840fb8","Type":"ContainerDied","Data":"8a3a67e6aefaaeeab3d9eb8ba058fd2f4eb8a17f68caae0a133c8b49d3c9bd1b"} Jul 31 20:30:26.171993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.171964 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" event={"ID":"9ceb4d3aec329af764e1753333df00b0","Type":"ContainerStarted","Data":"c3e425d9ad0401520c7f25f318c1ac30a2231debb0ebce83b20e12aeaccfd159"} Jul 31 20:30:26.176195 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.176167 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2tm82" event={"ID":"2dda82ab-5555-4f13-b159-df10a22d5b36","Type":"ContainerStarted","Data":"a7ecf02320831938785c24d7b01a2430115aa32e14dffe90d2f8f4ff933aced1"} Jul 31 20:30:26.179054 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.179025 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8hvl" event={"ID":"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3","Type":"ContainerStarted","Data":"8296829b33d1b620a736cdcfdec5e73cf02169373a93eb5400c870ccbd03c7ba"} Jul 31 20:30:26.184648 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.184622 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t94kz" event={"ID":"c6a651d5-2e0e-490d-95b5-15f7165eba9f","Type":"ContainerStarted","Data":"8b3d3f924304a5bed4332568774361e59e965dda8d5ac55616a1f452f3b089ee"} Jul 31 20:30:26.194259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:26.194062 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x89qb" event={"ID":"9027ab96-d2cc-474c-b92d-9865ba493785","Type":"ContainerStarted","Data":"a15084bf5a8a53f181b6b8c46a9b67e335ed4c0159a3a4ee9179d59bc9df303a"} Jul 31 20:30:27.222466 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:27.222252 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" event={"ID":"fc51ba05675423a991fa65b5ef840fb8","Type":"ContainerStarted","Data":"5d59875aba63a9a3b5c09852ffd5a1098a97e8913981f8ef4d78ce43395741a0"} Jul 31 20:30:27.238994 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:27.238935 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-177.ec2.internal" podStartSLOduration=4.23890276 podStartE2EDuration="4.23890276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:26.19236691 +0000 UTC m=+4.568740089" watchObservedRunningTime="2026-07-31 20:30:27.23890276 +0000 UTC m=+5.615275941" Jul 31 20:30:27.723696 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:27.723656 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:27.723933 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:27.723705 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:27.723933 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.723884 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:27.724073 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.723947 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:31.723926867 +0000 UTC m=+10.100300026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:27.724323 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.724226 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:27.724323 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.724284 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:31.724266232 +0000 UTC m=+10.100639404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:27.825120 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:27.825082 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:27.825364 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.825293 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:27.825364 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.825319 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:27.825364 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.825330 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:27.825529 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:27.825387 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:31.825368196 +0000 UTC m=+10.201741355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:28.109292 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:28.109257 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:28.109497 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:28.109407 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:28.109827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:28.109806 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:28.109952 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:28.109929 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:28.110019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:28.110005 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:28.110337 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:28.110082 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:30.107634 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:30.107600 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:30.108173 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:30.107760 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:30.108173 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:30.107991 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:30.108173 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:30.108110 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:30.108336 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:30.108265 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:30.108336 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:30.108330 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:31.759090 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:31.759034 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:31.759090 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:31.759077 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:31.759627 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.759215 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:31.759627 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.759216 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:31.759627 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.759290 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:39.759269204 +0000 UTC m=+18.135642384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:31.759627 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.759308 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:39.75929946 +0000 UTC m=+18.135672623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:31.860075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:31.860035 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:31.860323 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.860242 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:31.860323 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.860264 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:31.860323 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.860276 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:31.860480 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:31.860334 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:39.86031563 +0000 UTC m=+18.236688791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:32.111701 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:32.111667 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:32.111889 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:32.111796 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:32.112472 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:32.112190 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:32.112472 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:32.112282 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:32.112472 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:32.112338 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:32.112472 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:32.112409 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:34.108752 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:34.108887 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:34.108961 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:34.109040 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:34.109089 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:34.109246 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:34.109165 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:36.107158 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:36.107123 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:36.107624 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:36.107167 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:36.107624 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:36.107127 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:36.107624 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:36.107231 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:36.107624 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:36.107305 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:36.107624 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:36.107370 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:38.107008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:38.106799 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:38.107008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:38.106855 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:38.107008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:38.106820 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:38.107008 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:38.106964 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:38.107561 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:38.107055 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:38.107561 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:38.107136 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:39.823580 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:39.823536 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:39.824169 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:39.823587 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:39.824169 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.823713 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:39.824169 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.823719 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:39.824169 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.823780 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:55.823763896 +0000 UTC m=+34.200137071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:39.824169 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.823801 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:30:55.823790463 +0000 UTC m=+34.200163623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:39.924114 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:39.924075 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:39.924298 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.924228 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:39.924298 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.924257 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:39.924298 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.924269 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:39.924445 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:39.924337 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:30:55.924317659 +0000 UTC m=+34.300690837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:40.107089 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:40.107059 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:40.107278 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:40.107073 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:40.107278 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:40.107195 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:40.107278 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:40.107242 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:40.107438 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:40.107318 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:40.107438 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:40.107375 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:42.107604 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:42.107568 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:42.108086 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:42.107675 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:42.108086 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:42.107729 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:42.108086 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:42.107828 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:42.108086 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:42.107890 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:42.108086 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:42.107958 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:44.107390 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.107038 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:44.107764 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:44.107567 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:44.108086 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.108062 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:44.108191 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:44.108167 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:44.110456 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.110430 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:44.110559 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:44.110534 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:44.262455 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.262247 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2tm82" event={"ID":"2dda82ab-5555-4f13-b159-df10a22d5b36","Type":"ContainerStarted","Data":"e2721ceb154e12a7fa0383ca62c32daeea1272a9764c48ded6585cf43f2d4560"} Jul 31 20:30:44.264425 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.264397 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8hvl" event={"ID":"c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3","Type":"ContainerStarted","Data":"736583f2c9b9b5d1f77d94dfc4fabdd9192f6fd349e6f49cd8323d35ae018510"} Jul 31 20:30:44.266286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.266256 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t94kz" event={"ID":"c6a651d5-2e0e-490d-95b5-15f7165eba9f","Type":"ContainerStarted","Data":"2063bec22bf97dfe373be04b8d2bc78d9d5f63750c0357931b8006005f104b12"} Jul 31 20:30:44.268742 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.268713 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x89qb" event={"ID":"9027ab96-d2cc-474c-b92d-9865ba493785","Type":"ContainerStarted","Data":"9459f62c42f2a279cc052d2246512bcc2d4754342491e47d59b487f4ba449e33"} Jul 31 20:30:44.270403 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.270378 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" event={"ID":"72af3523-18bb-4a16-a0d4-b6db4881ce46","Type":"ContainerStarted","Data":"082b0c94f21b2b3c7045970ec05fa5d324839f2321fa35ed1aa1c3f95e1e9cd8"} Jul 31 20:30:44.271930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.271900 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68whh" event={"ID":"585a0b01-8e0b-4f08-a98e-33508e42d0bd","Type":"ContainerStarted","Data":"f1f77ee292bdf8cdd0baef9897a255eefb1214028384e50f8fa24d9d93cf60cd"} Jul 31 20:30:44.276121 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.276091 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerStarted","Data":"1d840c454a39c16990c270f8169120d30f57dc3df611c9d98167c01e3f6d7785"} Jul 31 20:30:44.277747 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.277723 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"85510cbe3098371e1f53c600af8ed159073389b30aaa0504406d3edcecdd483c"} Jul 31 20:30:44.277846 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.277778 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"5c455b6d4be53979d6f2af3222c734b7fc3400dc053ad8c7e910edf6266f8f7f"} Jul 31 20:30:44.282152 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.282099 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-177.ec2.internal" podStartSLOduration=21.282085677 podStartE2EDuration="21.282085677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:27.239542445 +0000 UTC m=+5.615915625" watchObservedRunningTime="2026-07-31 20:30:44.282085677 +0000 UTC m=+22.658458902" Jul 31 20:30:44.282286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.282183 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2tm82" podStartSLOduration=3.412432631 podStartE2EDuration="22.282177798s" totalImagesPullingTime="18.869745167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:44.281798749 +0000 UTC m=+22.658171928" watchObservedRunningTime="2026-07-31 20:30:44.282177798 +0000 UTC m=+22.658550977" Jul 31 20:30:44.296767 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.296715 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v8hvl" podStartSLOduration=3.488237797 podStartE2EDuration="22.296700467s" totalImagesPullingTime="18.80846267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:44.296440325 +0000 UTC m=+22.672813503" watchObservedRunningTime="2026-07-31 20:30:44.296700467 +0000 UTC m=+22.673073645" Jul 31 20:30:44.309943 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.309854 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x89qb" podStartSLOduration=3.481079863 podStartE2EDuration="22.309839672s" totalImagesPullingTime="18.828759809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:44.309427433 +0000 UTC m=+22.685800614" watchObservedRunningTime="2026-07-31 20:30:44.309839672 +0000 UTC m=+22.686212850" Jul 31 20:30:44.346100 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.346053 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t94kz" podStartSLOduration=3.514429382 podStartE2EDuration="22.346037197s" totalImagesPullingTime="18.831607815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:44.345279703 +0000 UTC m=+22.721652882" watchObservedRunningTime="2026-07-31 20:30:44.346037197 +0000 UTC m=+22.722410376" Jul 31 20:30:44.361161 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:44.361119 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-68whh" podStartSLOduration=3.554872766 podStartE2EDuration="22.361103397s" totalImagesPullingTime="18.806230631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:44.360792089 +0000 UTC m=+22.737165268" watchObservedRunningTime="2026-07-31 20:30:44.361103397 +0000 UTC m=+22.737476576" Jul 31 20:30:45.281515 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.281480 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="1d840c454a39c16990c270f8169120d30f57dc3df611c9d98167c01e3f6d7785" exitCode=0 Jul 31 20:30:45.282269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.281528 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"1d840c454a39c16990c270f8169120d30f57dc3df611c9d98167c01e3f6d7785"} Jul 31 20:30:45.283098 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.283077 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxvp5" event={"ID":"eb64b08a-e299-4eba-9311-0126eb3f0792","Type":"ContainerStarted","Data":"7dc5a08d4d0fd51cbf6347650f3f9a1f583fc0f03b91b68a3319943d335dd97d"} Jul 31 20:30:45.286214 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.286186 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"786da6eba5d19401c63d12803fe6e8d1d1d66f1d7a93cef8c44f7bb2663b2563"} Jul 31 20:30:45.286330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.286252 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"4928c12d6c6118179d09041ef8503b185279db3021ea5ab1805bd4804ad9f028"} Jul 31 20:30:45.286330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.286288 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"f00947c75533fe421cba0f009940954630a5085b1a8ab440f28f28e28b5f3079"} Jul 31 20:30:45.286399 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.286357 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"d2390e62113aedbafe644177ca2a35521a7f2066c047027ad48d7ece46c11757"} Jul 31 20:30:45.556598 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:45.556559 2544 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 20:30:46.034133 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.034026 2544 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T20:30:45.556592307Z","UUID":"4ec3fef8-4aa9-4347-a2ea-3debac4048f9","Handler":null,"Name":"","Endpoint":""} Jul 31 20:30:46.035519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.035498 2544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 20:30:46.035654 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.035536 2544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 20:30:46.107227 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.107198 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:46.107405 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.107202 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:46.107405 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:46.107334 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:46.107405 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.107399 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:46.107556 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:46.107509 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:46.107613 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:46.107591 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:46.290772 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.290680 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" event={"ID":"72af3523-18bb-4a16-a0d4-b6db4881ce46","Type":"ContainerStarted","Data":"b8991c2932b91913ba10d87d8eb9e2f45ce38d727df752649e744ec5077e2c97"} Jul 31 20:30:46.306437 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:46.306389 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bxvp5" podStartSLOduration=5.490036468 podStartE2EDuration="24.30637413s" totalImagesPullingTime="18.816337662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:46.306336815 +0000 UTC m=+24.682709994" watchObservedRunningTime="2026-07-31 20:30:46.30637413 +0000 UTC m=+24.682747305" Jul 31 20:30:47.295470 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:47.295227 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" event={"ID":"72af3523-18bb-4a16-a0d4-b6db4881ce46","Type":"ContainerStarted","Data":"503a1e92637526fe447d17f4c917df85bbe5e6c8c9fb67b5f37ce65635187480"} Jul 31 20:30:47.299068 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:47.299023 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"4ef740fdc3246791511c80b3ef4796a496cb444b086a51df726cc990e2a47256"} Jul 31 20:30:47.312419 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:47.312368 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gr6tc" podStartSLOduration=3.956204843 podStartE2EDuration="25.312345541s" totalImagesPullingTime="21.356140698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:47.311086363 +0000 UTC m=+25.687459541" watchObservedRunningTime="2026-07-31 20:30:47.312345541 +0000 UTC m=+25.688718717" Jul 31 20:30:48.107215 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.107184 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:48.107414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.107181 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:48.107414 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:48.107341 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:48.107414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.107181 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:48.107414 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:48.107387 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:48.107606 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:48.107465 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:48.494318 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.494226 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:48.494816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.494797 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:48.984504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.984452 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:48.985180 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:48.985161 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x89qb" Jul 31 20:30:50.107300 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:50.107116 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:50.107674 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:50.107116 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:50.107674 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:50.107379 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:50.107674 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:50.107120 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:50.107674 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:50.107462 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:50.107674 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:50.107565 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:50.310527 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:50.310499 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" event={"ID":"fafcf7c3-be67-4f67-a14b-6147b5845d1e","Type":"ContainerStarted","Data":"dd06610b1e856b5a14bcd27b173e32b302b0a60d8551ea74fe5adfeeaf6e02ee"} Jul 31 20:30:50.335408 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:50.335350 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" podStartSLOduration=9.431216735 podStartE2EDuration="28.335328977s" totalImagesPullingTime="18.904112242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:30:50.333555785 +0000 UTC m=+28.709928968" watchObservedRunningTime="2026-07-31 20:30:50.335328977 +0000 UTC m=+28.711702158" Jul 31 20:30:51.314433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.314398 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="9c5aed50be5c6172793dcf0bac0d3e4c3cd44cb02b938d41b91370452ce3d273" exitCode=0 Jul 31 20:30:51.315283 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.314490 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"9c5aed50be5c6172793dcf0bac0d3e4c3cd44cb02b938d41b91370452ce3d273"} Jul 31 20:30:51.315283 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.314896 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:51.315283 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.314971 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:51.315283 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.315016 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:51.331917 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.331707 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:51.332183 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:51.332025 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:30:52.108620 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:52.108585 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:52.108804 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:52.108672 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:52.108804 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:52.108714 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:52.108804 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:52.108753 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:52.108804 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:52.108786 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:52.109014 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:52.108888 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:52.318246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:52.318215 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="21996f0c1f15af69e49c700992fac9f5d00712f53d504180f0f7cc42b677b66e" exitCode=0 Jul 31 20:30:52.318646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:52.318294 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"21996f0c1f15af69e49c700992fac9f5d00712f53d504180f0f7cc42b677b66e"} Jul 31 20:30:53.322571 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:53.322476 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="9941239634a6aacf8c73e4cf20dabf846cce287875f610548c82446d8aeb14e0" exitCode=0 Jul 31 20:30:53.323054 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:53.322563 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"9941239634a6aacf8c73e4cf20dabf846cce287875f610548c82446d8aeb14e0"} Jul 31 20:30:54.107350 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:54.107315 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:54.107527 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:54.107319 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:54.107527 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:54.107452 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:54.107648 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:54.107540 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:54.107648 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:54.107319 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:54.107648 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:54.107640 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:55.832993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:55.832955 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:55.832993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:55.832997 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:55.833615 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.833115 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:55.833615 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.833115 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:55.833615 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.833186 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs podName:8ad42f23-7c17-4dd6-abdf-47febed6fa7b nodeName:}" failed. No retries permitted until 2026-07-31 20:31:27.833167105 +0000 UTC m=+66.209540262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs") pod "network-metrics-daemon-285jk" (UID: "8ad42f23-7c17-4dd6-abdf-47febed6fa7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 20:30:55.833615 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.833204 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret podName:99e6c527-9518-4a36-81c2-ea2634d4ecca nodeName:}" failed. No retries permitted until 2026-07-31 20:31:27.833195821 +0000 UTC m=+66.209568991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret") pod "global-pull-secret-syncer-b6gsd" (UID: "99e6c527-9518-4a36-81c2-ea2634d4ecca") : object "kube-system"/"original-pull-secret" not registered Jul 31 20:30:55.933986 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:55.933948 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:55.934190 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.934167 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 20:30:55.934260 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.934198 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 20:30:55.934260 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.934213 2544 projected.go:196] Error preparing data for projected volume kube-api-access-v9gxp for pod openshift-network-diagnostics/network-check-target-nm599: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:55.934359 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:55.934282 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp podName:049da32c-25fd-41a3-9e0d-fd9ee564d56b nodeName:}" failed. No retries permitted until 2026-07-31 20:31:27.934262108 +0000 UTC m=+66.310635269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9gxp" (UniqueName: "kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp") pod "network-check-target-nm599" (UID: "049da32c-25fd-41a3-9e0d-fd9ee564d56b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 20:30:56.106627 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:56.106590 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:56.106820 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:56.106590 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:56.106820 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:56.106701 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:56.106820 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:56.106590 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:56.106820 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:56.106793 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:56.107021 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:56.106938 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:58.106827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:58.106789 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:30:58.106827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:58.106806 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:30:58.106827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:58.106810 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:30:58.107457 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:58.106970 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:30:58.107457 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:58.107027 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:30:58.107457 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:30:58.107101 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:30:59.338176 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:59.338144 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="e8fc4cb9b7b1e350b76e02bb6d1510f22b1172fd6392053d82f1b804a7488897" exitCode=0 Jul 31 20:30:59.338581 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:30:59.338214 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"e8fc4cb9b7b1e350b76e02bb6d1510f22b1172fd6392053d82f1b804a7488897"} Jul 31 20:31:00.106826 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.106779 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:00.107034 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.106779 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:00.107034 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.106928 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:31:00.107034 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.106938 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:00.107034 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.107020 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:31:00.107254 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.107104 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:31:00.206018 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.205985 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nm599"] Jul 31 20:31:00.209110 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.209081 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-285jk"] Jul 31 20:31:00.220507 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.220430 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6gsd"] Jul 31 20:31:00.343737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.343701 2544 generic.go:356] "Generic (PLEG): container finished" podID="f92dc492-6985-4129-817c-60cbd61a0d6e" containerID="51b4bba9c3ae8bb10ffce2b05993fa675d5af1a2b237c84aa8eaf835251d03c1" exitCode=0 Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.343788 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.343819 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerDied","Data":"51b4bba9c3ae8bb10ffce2b05993fa675d5af1a2b237c84aa8eaf835251d03c1"} Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.343854 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:00.343958 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.343963 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.344078 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:31:00.344200 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:00.344130 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:31:01.349573 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:01.349384 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" event={"ID":"f92dc492-6985-4129-817c-60cbd61a0d6e","Type":"ContainerStarted","Data":"7fc4903587b96eff8a18faf10cfe21485fc213b9599f1f1bdeae82008807794f"} Jul 31 20:31:01.371466 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:01.371420 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g8cfg" podStartSLOduration=8.893266840999999 podStartE2EDuration="39.371402492s" totalImagesPullingTime="30.478135651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:01.369839765 +0000 UTC m=+39.746212944" watchObservedRunningTime="2026-07-31 20:31:01.371402492 +0000 UTC m=+39.747775670" Jul 31 20:31:02.108467 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:02.108429 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:02.108638 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:02.108524 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:02.108638 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:02.108566 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:02.108638 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:02.108560 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:31:02.108732 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:02.108649 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:31:02.108775 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:02.108745 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:31:04.106963 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:04.106934 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:04.106963 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:04.106960 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:04.107591 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:04.106935 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:04.107591 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:04.107040 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-285jk" podUID="8ad42f23-7c17-4dd6-abdf-47febed6fa7b" Jul 31 20:31:04.107591 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:04.107113 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6gsd" podUID="99e6c527-9518-4a36-81c2-ea2634d4ecca" Jul 31 20:31:04.107591 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:04.107178 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nm599" podUID="049da32c-25fd-41a3-9e0d-fd9ee564d56b" Jul 31 20:31:04.992616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:04.992587 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-177.ec2.internal" event="NodeReady" Jul 31 20:31:04.992790 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:04.992706 2544 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 20:31:05.036402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.036365 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fdttp"] Jul 31 20:31:05.057065 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.057031 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jxz27"] Jul 31 20:31:05.057213 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.057192 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.059635 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.059609 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 20:31:05.059787 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.059659 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w7w5c\"" Jul 31 20:31:05.059787 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.059609 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 20:31:05.081111 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.081078 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdttp"] Jul 31 20:31:05.081111 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.081117 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jxz27"] Jul 31 20:31:05.081271 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.081213 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.083664 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.083640 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 20:31:05.083899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.083685 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 20:31:05.083899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.083705 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdl8z\"" Jul 31 20:31:05.084095 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.083996 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 20:31:05.084095 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.084000 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 20:31:05.136207 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.136175 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gchrn"] Jul 31 20:31:05.160186 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.160159 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gchrn"] Jul 31 20:31:05.160347 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.160274 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.162880 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.162838 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 20:31:05.162880 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.162847 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 20:31:05.163082 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.162847 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 20:31:05.163082 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.162932 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jbv2m\"" Jul 31 20:31:05.202511 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202480 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglbx\" (UniqueName: \"kubernetes.io/projected/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-api-access-nglbx\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.202511 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202514 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/301dda6b-10e8-4f0e-be29-f4f520f6a061-config-volume\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.202696 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202530 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249rs\" (UniqueName: \"kubernetes.io/projected/301dda6b-10e8-4f0e-be29-f4f520f6a061-kube-api-access-249rs\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.202696 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202555 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb8484d7-b167-466e-8a4f-1c8c0a85534d-crio-socket\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.202696 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202596 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb8484d7-b167-466e-8a4f-1c8c0a85534d-data-volume\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.202696 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202687 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb8484d7-b167-466e-8a4f-1c8c0a85534d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.202809 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202709 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.202809 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202729 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/301dda6b-10e8-4f0e-be29-f4f520f6a061-metrics-tls\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.202809 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.202748 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301dda6b-10e8-4f0e-be29-f4f520f6a061-tmp-dir\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.303913 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.303813 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb8484d7-b167-466e-8a4f-1c8c0a85534d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.303913 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.303856 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.303913 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.303906 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/301dda6b-10e8-4f0e-be29-f4f520f6a061-metrics-tls\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.303928 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301dda6b-10e8-4f0e-be29-f4f520f6a061-tmp-dir\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.303947 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chncn\" (UniqueName: \"kubernetes.io/projected/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-kube-api-access-chncn\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304002 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglbx\" (UniqueName: \"kubernetes.io/projected/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-api-access-nglbx\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304069 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/301dda6b-10e8-4f0e-be29-f4f520f6a061-config-volume\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304105 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249rs\" (UniqueName: \"kubernetes.io/projected/301dda6b-10e8-4f0e-be29-f4f520f6a061-kube-api-access-249rs\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.304149 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304139 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb8484d7-b167-466e-8a4f-1c8c0a85534d-crio-socket\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.304402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304162 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb8484d7-b167-466e-8a4f-1c8c0a85534d-data-volume\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.304402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304190 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-cert\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.304402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304219 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/301dda6b-10e8-4f0e-be29-f4f520f6a061-tmp-dir\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.304563 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304432 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb8484d7-b167-466e-8a4f-1c8c0a85534d-crio-socket\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.304625 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.304606 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/301dda6b-10e8-4f0e-be29-f4f520f6a061-config-volume\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.308528 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.308512 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/301dda6b-10e8-4f0e-be29-f4f520f6a061-metrics-tls\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.312221 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.312199 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-249rs\" (UniqueName: \"kubernetes.io/projected/301dda6b-10e8-4f0e-be29-f4f520f6a061-kube-api-access-249rs\") pod \"dns-default-fdttp\" (UID: \"301dda6b-10e8-4f0e-be29-f4f520f6a061\") " pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.315250 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.315222 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb8484d7-b167-466e-8a4f-1c8c0a85534d-data-volume\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.315430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.315412 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.317249 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.317232 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb8484d7-b167-466e-8a4f-1c8c0a85534d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.317493 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.317473 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglbx\" (UniqueName: \"kubernetes.io/projected/cb8484d7-b167-466e-8a4f-1c8c0a85534d-kube-api-access-nglbx\") pod \"insights-runtime-extractor-jxz27\" (UID: \"cb8484d7-b167-466e-8a4f-1c8c0a85534d\") " pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.366302 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.366267 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:05.389644 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.389625 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jxz27" Jul 31 20:31:05.405055 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.405025 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chncn\" (UniqueName: \"kubernetes.io/projected/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-kube-api-access-chncn\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.405205 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.405118 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-cert\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.408002 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.407975 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-cert\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.414712 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.414468 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chncn\" (UniqueName: \"kubernetes.io/projected/0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40-kube-api-access-chncn\") pod \"ingress-canary-gchrn\" (UID: \"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40\") " pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.469426 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.469384 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gchrn" Jul 31 20:31:05.582134 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.582103 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jxz27"] Jul 31 20:31:05.584359 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.584333 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdttp"] Jul 31 20:31:05.585788 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:05.585763 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8484d7_b167_466e_8a4f_1c8c0a85534d.slice/crio-abdb0f4b2ded654033781bc2f68c3aab8043da02ad7d16e722fd4ad1e57cec85 WatchSource:0}: Error finding container abdb0f4b2ded654033781bc2f68c3aab8043da02ad7d16e722fd4ad1e57cec85: Status 404 returned error can't find the container with id abdb0f4b2ded654033781bc2f68c3aab8043da02ad7d16e722fd4ad1e57cec85 Jul 31 20:31:05.590255 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:05.588244 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301dda6b_10e8_4f0e_be29_f4f520f6a061.slice/crio-4be43bb7b0d760c9b185487c53756652c2cbf2ede9db946e2b4270ff63d2ce43 WatchSource:0}: Error finding container 4be43bb7b0d760c9b185487c53756652c2cbf2ede9db946e2b4270ff63d2ce43: Status 404 returned error can't find the container with id 4be43bb7b0d760c9b185487c53756652c2cbf2ede9db946e2b4270ff63d2ce43 Jul 31 20:31:05.638041 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:05.637802 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gchrn"] Jul 31 20:31:05.647500 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:05.647471 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac1a04c_44bc_4eb3_aefe_aae6f3c64f40.slice/crio-a787b4e014ff2d8cd64c873292dd599bf58fec34c6160b63c8e6843016d2f3fb WatchSource:0}: Error finding container a787b4e014ff2d8cd64c873292dd599bf58fec34c6160b63c8e6843016d2f3fb: Status 404 returned error can't find the container with id a787b4e014ff2d8cd64c873292dd599bf58fec34c6160b63c8e6843016d2f3fb Jul 31 20:31:06.107366 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.107335 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:06.108009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.107335 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:06.108009 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.107335 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:06.110127 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.110093 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 20:31:06.110291 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.110267 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 20:31:06.111257 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.111203 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 20:31:06.111361 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.111287 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6hmj\"" Jul 31 20:31:06.111361 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.111333 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 20:31:06.111493 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.111287 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9lfhf\"" Jul 31 20:31:06.365555 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.365459 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gchrn" event={"ID":"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40","Type":"ContainerStarted","Data":"a787b4e014ff2d8cd64c873292dd599bf58fec34c6160b63c8e6843016d2f3fb"} Jul 31 20:31:06.367641 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.367611 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxz27" event={"ID":"cb8484d7-b167-466e-8a4f-1c8c0a85534d","Type":"ContainerStarted","Data":"8051f3e768ac219eac1dbcdb387bde39e66c154bd6ac93a4d8f4d8be74a7c10f"} Jul 31 20:31:06.367791 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.367669 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxz27" event={"ID":"cb8484d7-b167-466e-8a4f-1c8c0a85534d","Type":"ContainerStarted","Data":"abdb0f4b2ded654033781bc2f68c3aab8043da02ad7d16e722fd4ad1e57cec85"} Jul 31 20:31:06.368965 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:06.368936 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdttp" event={"ID":"301dda6b-10e8-4f0e-be29-f4f520f6a061","Type":"ContainerStarted","Data":"4be43bb7b0d760c9b185487c53756652c2cbf2ede9db946e2b4270ff63d2ce43"} Jul 31 20:31:07.064882 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.064842 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4"] Jul 31 20:31:07.091090 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.091049 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4"] Jul 31 20:31:07.091271 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.091195 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:07.093624 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.093596 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 31 20:31:07.093624 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.093608 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rkhrp\"" Jul 31 20:31:07.223958 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.223920 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-s27h4\" (UID: \"691be1d0-d323-4627-b52a-03af75a442aa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:07.324498 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.324409 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-s27h4\" (UID: \"691be1d0-d323-4627-b52a-03af75a442aa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:07.324665 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:07.324494 2544 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 31 20:31:07.324665 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:07.324569 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates podName:691be1d0-d323-4627-b52a-03af75a442aa nodeName:}" failed. No retries permitted until 2026-07-31 20:31:07.824550812 +0000 UTC m=+46.200923972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-s27h4" (UID: "691be1d0-d323-4627-b52a-03af75a442aa") : secret "prometheus-operator-admission-webhook-tls" not found Jul 31 20:31:07.830286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.830247 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-s27h4\" (UID: \"691be1d0-d323-4627-b52a-03af75a442aa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:07.835025 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:07.834997 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/691be1d0-d323-4627-b52a-03af75a442aa-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-s27h4\" (UID: \"691be1d0-d323-4627-b52a-03af75a442aa\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:08.003311 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:08.003258 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:08.797787 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:08.791197 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4"] Jul 31 20:31:08.817166 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:08.817136 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691be1d0_d323_4627_b52a_03af75a442aa.slice/crio-595a13b0a8f1288e02e0ebe209879577a75f35236c500acf3f7897c53354afd6 WatchSource:0}: Error finding container 595a13b0a8f1288e02e0ebe209879577a75f35236c500acf3f7897c53354afd6: Status 404 returned error can't find the container with id 595a13b0a8f1288e02e0ebe209879577a75f35236c500acf3f7897c53354afd6 Jul 31 20:31:09.380257 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.380204 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdttp" event={"ID":"301dda6b-10e8-4f0e-be29-f4f520f6a061","Type":"ContainerStarted","Data":"11dd72a1bc81dd168c019a981abcabcc330cb495b085aae14aed3aeb50a5acce"} Jul 31 20:31:09.381230 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.380288 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdttp" event={"ID":"301dda6b-10e8-4f0e-be29-f4f520f6a061","Type":"ContainerStarted","Data":"c0dfdf556802c559635d5b37182b87563bfb19bc1ae8cbab7eca9c0cef10f845"} Jul 31 20:31:09.381230 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.380322 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:09.381951 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.381918 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gchrn" event={"ID":"0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40","Type":"ContainerStarted","Data":"f1c1d2de872d40437c71c8893340363032f7cc8802c01a6b57d83de3024acd72"} Jul 31 20:31:09.383943 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.383914 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxz27" event={"ID":"cb8484d7-b167-466e-8a4f-1c8c0a85534d","Type":"ContainerStarted","Data":"887d55a054d103ee9eb3a9e21a3d642d235e4535682c57b7089d839d1818989b"} Jul 31 20:31:09.385191 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.385166 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" event={"ID":"691be1d0-d323-4627-b52a-03af75a442aa","Type":"ContainerStarted","Data":"595a13b0a8f1288e02e0ebe209879577a75f35236c500acf3f7897c53354afd6"} Jul 31 20:31:09.397076 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.397020 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fdttp" podStartSLOduration=1.3364712650000001 podStartE2EDuration="4.397001445s" totalImagesPullingTime="3.06053018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:09.394700708 +0000 UTC m=+47.771073888" watchObservedRunningTime="2026-07-31 20:31:09.397001445 +0000 UTC m=+47.773374625" Jul 31 20:31:09.409022 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:09.408971 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gchrn" podStartSLOduration=1.399298028 podStartE2EDuration="4.408950059s" totalImagesPullingTime="3.009652031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:09.40872896 +0000 UTC m=+47.785102140" watchObservedRunningTime="2026-07-31 20:31:09.408950059 +0000 UTC m=+47.785323241" Jul 31 20:31:11.393519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.393261 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jxz27" event={"ID":"cb8484d7-b167-466e-8a4f-1c8c0a85534d","Type":"ContainerStarted","Data":"7c2395ddc9d65bbb73e36c60d6bbe8f2e695ae162994d7ae1408082fe8b016b6"} Jul 31 20:31:11.394917 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.394884 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" event={"ID":"691be1d0-d323-4627-b52a-03af75a442aa","Type":"ContainerStarted","Data":"340d78c75190ee8c00d49a8e3cb0afc900d1b31654ff3a8c3379604fa8151fd5"} Jul 31 20:31:11.395109 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.395090 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:11.400005 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.399982 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" Jul 31 20:31:11.410728 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.410685 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jxz27" podStartSLOduration=1.36725855 podStartE2EDuration="6.410671144s" totalImagesPullingTime="5.043412594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:31:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:11.409956499 +0000 UTC m=+49.786329680" watchObservedRunningTime="2026-07-31 20:31:11.410671144 +0000 UTC m=+49.787044300" Jul 31 20:31:11.425851 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:11.425803 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-s27h4" podStartSLOduration=2.338363978 podStartE2EDuration="4.425790372s" totalImagesPullingTime="2.087426394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:11.425584782 +0000 UTC m=+49.801957962" watchObservedRunningTime="2026-07-31 20:31:11.425790372 +0000 UTC m=+49.802163561" Jul 31 20:31:12.119120 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.119087 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-d9rwz"] Jul 31 20:31:12.134433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.134401 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.137161 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.137137 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-d9rwz"] Jul 31 20:31:12.137675 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.137655 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 20:31:12.138590 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.138509 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-65cpd\"" Jul 31 20:31:12.138590 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.138539 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 20:31:12.138590 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.138556 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 31 20:31:12.138818 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.138559 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 31 20:31:12.138818 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.138510 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 20:31:12.167504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.167458 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.167719 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.167531 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvns\" (UniqueName: \"kubernetes.io/projected/01616c7a-ffbe-42da-89ea-b24e372363bc-kube-api-access-drvns\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.167719 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.167564 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.167719 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.167601 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01616c7a-ffbe-42da-89ea-b24e372363bc-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.268029 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.267993 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvns\" (UniqueName: \"kubernetes.io/projected/01616c7a-ffbe-42da-89ea-b24e372363bc-kube-api-access-drvns\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.268191 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.268040 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.268191 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.268079 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01616c7a-ffbe-42da-89ea-b24e372363bc-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.268191 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.268109 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.268287 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:12.268197 2544 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 31 20:31:12.268287 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:12.268259 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls podName:01616c7a-ffbe-42da-89ea-b24e372363bc nodeName:}" failed. No retries permitted until 2026-07-31 20:31:12.768240988 +0000 UTC m=+51.144614147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-d9rwz" (UID: "01616c7a-ffbe-42da-89ea-b24e372363bc") : secret "prometheus-operator-tls" not found Jul 31 20:31:12.268811 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.268790 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01616c7a-ffbe-42da-89ea-b24e372363bc-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.272235 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.272211 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.278463 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.278439 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvns\" (UniqueName: \"kubernetes.io/projected/01616c7a-ffbe-42da-89ea-b24e372363bc-kube-api-access-drvns\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.771483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:12.771449 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:12.771949 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:12.771571 2544 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 31 20:31:12.771949 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:12.771637 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls podName:01616c7a-ffbe-42da-89ea-b24e372363bc nodeName:}" failed. No retries permitted until 2026-07-31 20:31:13.77162218 +0000 UTC m=+52.147995338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-d9rwz" (UID: "01616c7a-ffbe-42da-89ea-b24e372363bc") : secret "prometheus-operator-tls" not found Jul 31 20:31:13.776092 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:13.776051 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:13.778710 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:13.778684 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/01616c7a-ffbe-42da-89ea-b24e372363bc-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-d9rwz\" (UID: \"01616c7a-ffbe-42da-89ea-b24e372363bc\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:13.944817 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:13.944776 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" Jul 31 20:31:14.092847 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:14.092670 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-d9rwz"] Jul 31 20:31:14.096842 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:14.096811 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01616c7a_ffbe_42da_89ea_b24e372363bc.slice/crio-98efd9df5e677c3671d249826caea19e32daac5d5ac2f93b756292b1d07d4514 WatchSource:0}: Error finding container 98efd9df5e677c3671d249826caea19e32daac5d5ac2f93b756292b1d07d4514: Status 404 returned error can't find the container with id 98efd9df5e677c3671d249826caea19e32daac5d5ac2f93b756292b1d07d4514 Jul 31 20:31:14.405764 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:14.405725 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" event={"ID":"01616c7a-ffbe-42da-89ea-b24e372363bc","Type":"ContainerStarted","Data":"98efd9df5e677c3671d249826caea19e32daac5d5ac2f93b756292b1d07d4514"} Jul 31 20:31:16.412837 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:16.412800 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" event={"ID":"01616c7a-ffbe-42da-89ea-b24e372363bc","Type":"ContainerStarted","Data":"20992fc088cdd6bde46770528bb2f6e897eee9ae6e8823bea8c568709e8c22c7"} Jul 31 20:31:16.413545 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:16.412851 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" event={"ID":"01616c7a-ffbe-42da-89ea-b24e372363bc","Type":"ContainerStarted","Data":"be6d8175f1a9a69c22780a4efe741095aedfc531fc6ff73d639c7aa77986a644"} Jul 31 20:31:16.437007 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:16.436956 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-d9rwz" podStartSLOduration=2.8837512050000003 podStartE2EDuration="4.436940479s" totalImagesPullingTime="1.553189274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:16.436010165 +0000 UTC m=+54.812383367" watchObservedRunningTime="2026-07-31 20:31:16.436940479 +0000 UTC m=+54.813313657" Jul 31 20:31:18.458220 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.458185 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx"] Jul 31 20:31:18.460239 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.460223 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.462705 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.462682 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 31 20:31:18.463035 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.463016 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q6tjs\"" Jul 31 20:31:18.463145 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.463023 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 31 20:31:18.473447 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.473419 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx"] Jul 31 20:31:18.484747 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.484717 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh"] Jul 31 20:31:18.486573 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.486558 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.489600 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.489575 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 31 20:31:18.489725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.489708 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 31 20:31:18.489981 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.489961 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4pgmg\"" Jul 31 20:31:18.490588 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.490571 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 31 20:31:18.492941 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.492922 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fh2qj"] Jul 31 20:31:18.494944 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.494927 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.497297 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.497269 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 20:31:18.497297 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.497296 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 20:31:18.497486 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.497353 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mbphr\"" Jul 31 20:31:18.497604 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.497575 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 20:31:18.501675 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.501655 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh"] Jul 31 20:31:18.507620 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507591 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a9a01394-3275-4296-baee-b1ac06a37452-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.507751 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507633 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.507816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507791 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrfx\" (UniqueName: \"kubernetes.io/projected/3dce5282-17a5-497e-a8dc-15e9c8cb8800-kube-api-access-msrfx\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.507897 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507832 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-metrics-client-ca\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.507957 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507858 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.507957 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507944 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.508052 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.507977 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.508052 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508001 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-textfile\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508052 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508017 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508047 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcj2l\" (UniqueName: \"kubernetes.io/projected/a9a01394-3275-4296-baee-b1ac06a37452-kube-api-access-qcj2l\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.508264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508080 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dce5282-17a5-497e-a8dc-15e9c8cb8800-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.508264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508111 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.508264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508144 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.508264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508221 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.508503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508278 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9gt\" (UniqueName: \"kubernetes.io/projected/3a4c2b8b-926d-416c-bbc3-5af77737ed39-kube-api-access-8r9gt\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508306 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-root\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508330 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-wtmp\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508387 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-sys\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.508503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.508429 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.609068 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609031 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-sys\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609068 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609076 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609148 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a9a01394-3275-4296-baee-b1ac06a37452-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609173 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609146 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-sys\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609199 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrfx\" (UniqueName: \"kubernetes.io/projected/3dce5282-17a5-497e-a8dc-15e9c8cb8800-kube-api-access-msrfx\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:18.609236 2544 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609246 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-metrics-client-ca\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:18.609268 2544 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609275 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609330 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:18.609318 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls podName:3a4c2b8b-926d-416c-bbc3-5af77737ed39 nodeName:}" failed. No retries permitted until 2026-07-31 20:31:19.109297222 +0000 UTC m=+57.485670404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls") pod "node-exporter-fh2qj" (UID: "3a4c2b8b-926d-416c-bbc3-5af77737ed39") : secret "node-exporter-tls" not found Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:18.609342 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls podName:3dce5282-17a5-497e-a8dc-15e9c8cb8800 nodeName:}" failed. No retries permitted until 2026-07-31 20:31:19.109333413 +0000 UTC m=+57.485706571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-knvzx" (UID: "3dce5282-17a5-497e-a8dc-15e9c8cb8800") : secret "openshift-state-metrics-tls" not found Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609362 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609397 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609422 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-textfile\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609447 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609476 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcj2l\" (UniqueName: \"kubernetes.io/projected/a9a01394-3275-4296-baee-b1ac06a37452-kube-api-access-qcj2l\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609507 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dce5282-17a5-497e-a8dc-15e9c8cb8800-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609536 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609574 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609597 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a9a01394-3275-4296-baee-b1ac06a37452-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609600 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609666 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9gt\" (UniqueName: \"kubernetes.io/projected/3a4c2b8b-926d-416c-bbc3-5af77737ed39-kube-api-access-8r9gt\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609693 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-root\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.609768 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609717 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-wtmp\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.609893 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-wtmp\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610025 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610028 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c2b8b-926d-416c-bbc3-5af77737ed39-metrics-client-ca\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610089 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c2b8b-926d-416c-bbc3-5af77737ed39-root\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610248 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610254 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a01394-3275-4296-baee-b1ac06a37452-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.610562 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610479 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-textfile\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.610807 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.610680 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dce5282-17a5-497e-a8dc-15e9c8cb8800-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.613343 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.613318 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.613454 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.613408 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.613454 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.613432 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a01394-3275-4296-baee-b1ac06a37452-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.613626 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.613607 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.617354 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.617324 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcj2l\" (UniqueName: \"kubernetes.io/projected/a9a01394-3275-4296-baee-b1ac06a37452-kube-api-access-qcj2l\") pod \"kube-state-metrics-6bd9664c4-hhprh\" (UID: \"a9a01394-3275-4296-baee-b1ac06a37452\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.617446 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.617324 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9gt\" (UniqueName: \"kubernetes.io/projected/3a4c2b8b-926d-416c-bbc3-5af77737ed39-kube-api-access-8r9gt\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:18.617446 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.617398 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrfx\" (UniqueName: \"kubernetes.io/projected/3dce5282-17a5-497e-a8dc-15e9c8cb8800-kube-api-access-msrfx\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:18.796319 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.796235 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" Jul 31 20:31:18.932525 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:18.932493 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh"] Jul 31 20:31:18.935578 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:18.935538 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a01394_3275_4296_baee_b1ac06a37452.slice/crio-06ca1319205470a6ddc66e01b98602278198162a1912c8839ba5f9a2eb0b132d WatchSource:0}: Error finding container 06ca1319205470a6ddc66e01b98602278198162a1912c8839ba5f9a2eb0b132d: Status 404 returned error can't find the container with id 06ca1319205470a6ddc66e01b98602278198162a1912c8839ba5f9a2eb0b132d Jul 31 20:31:19.113929 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.113890 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:19.114099 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.113959 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:19.116352 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.116325 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c2b8b-926d-416c-bbc3-5af77737ed39-node-exporter-tls\") pod \"node-exporter-fh2qj\" (UID: \"3a4c2b8b-926d-416c-bbc3-5af77737ed39\") " pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:19.116517 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.116498 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dce5282-17a5-497e-a8dc-15e9c8cb8800-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-knvzx\" (UID: \"3dce5282-17a5-497e-a8dc-15e9c8cb8800\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:19.369647 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.369537 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" Jul 31 20:31:19.390941 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.390895 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fdttp" Jul 31 20:31:19.405842 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.405808 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh2qj" Jul 31 20:31:19.417716 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:19.417684 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4c2b8b_926d_416c_bbc3_5af77737ed39.slice/crio-4ddf1efbd27084eed93751b25107b66c0215f566fb8684a2da2bb9db6bb9f2ec WatchSource:0}: Error finding container 4ddf1efbd27084eed93751b25107b66c0215f566fb8684a2da2bb9db6bb9f2ec: Status 404 returned error can't find the container with id 4ddf1efbd27084eed93751b25107b66c0215f566fb8684a2da2bb9db6bb9f2ec Jul 31 20:31:19.428302 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.428213 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh2qj" event={"ID":"3a4c2b8b-926d-416c-bbc3-5af77737ed39","Type":"ContainerStarted","Data":"4ddf1efbd27084eed93751b25107b66c0215f566fb8684a2da2bb9db6bb9f2ec"} Jul 31 20:31:19.430023 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.429939 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" event={"ID":"a9a01394-3275-4296-baee-b1ac06a37452","Type":"ContainerStarted","Data":"06ca1319205470a6ddc66e01b98602278198162a1912c8839ba5f9a2eb0b132d"} Jul 31 20:31:19.517595 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.517562 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx"] Jul 31 20:31:19.520703 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:19.520671 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dce5282_17a5_497e_a8dc_15e9c8cb8800.slice/crio-2a03201c50cc99e016052739fe94b224bc1d73407387db295abe8a727f235c6d WatchSource:0}: Error finding container 2a03201c50cc99e016052739fe94b224bc1d73407387db295abe8a727f235c6d: Status 404 returned error can't find the container with id 2a03201c50cc99e016052739fe94b224bc1d73407387db295abe8a727f235c6d Jul 31 20:31:19.542271 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.542246 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:31:19.546767 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.546745 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.549302 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549278 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 20:31:19.549434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549321 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 20:31:19.549434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549334 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 20:31:19.549434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549390 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 20:31:19.549625 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549611 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 20:31:19.549705 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549682 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 20:31:19.549763 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549751 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 20:31:19.549816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549752 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 20:31:19.549816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.549792 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6df5p\"" Jul 31 20:31:19.550049 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.550030 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 20:31:19.559184 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.559160 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:31:19.618724 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618688 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.618920 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618736 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.618920 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618799 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.618920 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618829 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.618920 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618861 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618924 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618957 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.618984 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.619027 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.619064 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.619111 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619135 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.619135 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.619474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.619165 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsqj\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.719785 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719743 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.719785 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719790 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719822 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719846 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719912 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719943 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.719987 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720009 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720034 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsqj\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720075 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720067 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720446 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720091 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720446 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720145 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720446 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720180 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.720606 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.720491 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.721276 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.721052 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.721412 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.721348 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.721479 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:19.721435 2544 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 31 20:31:19.721535 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:19.721494 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls podName:945a3d3d-2b11-4357-8e8a-bad63aef0394 nodeName:}" failed. No retries permitted until 2026-07-31 20:31:20.22147567 +0000 UTC m=+58.597848826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394") : secret "alertmanager-main-tls" not found Jul 31 20:31:19.723233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.723198 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.723403 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.723381 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.723882 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.723829 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.724087 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.724058 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.724257 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.724241 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.724317 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.724267 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.724317 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.724264 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.724430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.724415 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:19.729519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:19.729496 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsqj\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:20.005755 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.005664 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-fx8pm"] Jul 31 20:31:20.008552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.008525 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:20.011350 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.011321 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 20:31:20.011584 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.011565 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 20:31:20.011963 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.011911 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gcf5n\"" Jul 31 20:31:20.020502 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.020476 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-fx8pm"] Jul 31 20:31:20.022730 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.022705 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbdd\" (UniqueName: \"kubernetes.io/projected/836b7c03-2d71-4d64-a018-a2f319e82592-kube-api-access-4gbdd\") pod \"downloads-77c974ffcf-fx8pm\" (UID: \"836b7c03-2d71-4d64-a018-a2f319e82592\") " pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:20.124064 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.124031 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbdd\" (UniqueName: \"kubernetes.io/projected/836b7c03-2d71-4d64-a018-a2f319e82592-kube-api-access-4gbdd\") pod \"downloads-77c974ffcf-fx8pm\" (UID: \"836b7c03-2d71-4d64-a018-a2f319e82592\") " pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:20.132444 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.132411 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbdd\" (UniqueName: \"kubernetes.io/projected/836b7c03-2d71-4d64-a018-a2f319e82592-kube-api-access-4gbdd\") pod \"downloads-77c974ffcf-fx8pm\" (UID: \"836b7c03-2d71-4d64-a018-a2f319e82592\") " pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:20.224861 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.224817 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:20.227822 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.227794 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:20.321273 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.321007 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:20.438198 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.438154 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" event={"ID":"3dce5282-17a5-497e-a8dc-15e9c8cb8800","Type":"ContainerStarted","Data":"d5b19360478966950bfa8d3dd85df910ef396971a87f1af466ce5db8e85b8325"} Jul 31 20:31:20.438385 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.438233 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" event={"ID":"3dce5282-17a5-497e-a8dc-15e9c8cb8800","Type":"ContainerStarted","Data":"a06fc23b66e148753b87ef08e4df0b085b56353b6724b094f23fe57e2457ff50"} Jul 31 20:31:20.438385 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.438265 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" event={"ID":"3dce5282-17a5-497e-a8dc-15e9c8cb8800","Type":"ContainerStarted","Data":"2a03201c50cc99e016052739fe94b224bc1d73407387db295abe8a727f235c6d"} Jul 31 20:31:20.474519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.474482 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:31:20.485802 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.484730 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7"] Jul 31 20:31:20.487083 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.487065 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.490485 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.490460 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aq8gv6v1olple\"" Jul 31 20:31:20.490617 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.490598 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 31 20:31:20.490720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.490701 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 31 20:31:20.493881 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.493844 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6f9ts\"" Jul 31 20:31:20.493989 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.493955 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 31 20:31:20.494066 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.493957 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 31 20:31:20.494661 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.494491 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 31 20:31:20.517809 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.517777 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7"] Jul 31 20:31:20.527536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527501 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-grpc-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527574 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f62f997-53ea-474f-beaa-5244f0d1470d-metrics-client-ca\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527624 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527714 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swz4\" (UniqueName: \"kubernetes.io/projected/1f62f997-53ea-474f-beaa-5244f0d1470d-kube-api-access-9swz4\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527783 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527829 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527983 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527856 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.527983 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.527945 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.628981 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.628940 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swz4\" (UniqueName: \"kubernetes.io/projected/1f62f997-53ea-474f-beaa-5244f0d1470d-kube-api-access-9swz4\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629171 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629014 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629171 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629078 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629171 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629112 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629345 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629323 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629410 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629392 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-grpc-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629473 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629457 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f62f997-53ea-474f-beaa-5244f0d1470d-metrics-client-ca\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.629529 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.629517 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.630401 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.630374 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f62f997-53ea-474f-beaa-5244f0d1470d-metrics-client-ca\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.632603 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.632576 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.632704 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.632574 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.632780 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.632754 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.632952 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.632928 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-grpc-tls\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.633048 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.633011 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.633272 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.633256 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1f62f997-53ea-474f-beaa-5244f0d1470d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.642310 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.642290 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swz4\" (UniqueName: \"kubernetes.io/projected/1f62f997-53ea-474f-beaa-5244f0d1470d-kube-api-access-9swz4\") pod \"thanos-querier-6f5d9b6d48-4kfg7\" (UID: \"1f62f997-53ea-474f-beaa-5244f0d1470d\") " pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:20.797850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:20.797810 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:21.623414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:21.623380 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-fx8pm"] Jul 31 20:31:21.851142 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:21.851105 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7"] Jul 31 20:31:21.855072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:21.855045 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:31:21.856364 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:21.856299 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f62f997_53ea_474f_beaa_5244f0d1470d.slice/crio-09a36766f300559e72ce979257987b6279cb09fd389ca389da16bdc6297aa879 WatchSource:0}: Error finding container 09a36766f300559e72ce979257987b6279cb09fd389ca389da16bdc6297aa879: Status 404 returned error can't find the container with id 09a36766f300559e72ce979257987b6279cb09fd389ca389da16bdc6297aa879 Jul 31 20:31:21.861259 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:21.859825 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945a3d3d_2b11_4357_8e8a_bad63aef0394.slice/crio-e3cd50ebe5edd441ac4051340d6b82bb8e23154f91a44a7199a61a86a7574355 WatchSource:0}: Error finding container e3cd50ebe5edd441ac4051340d6b82bb8e23154f91a44a7199a61a86a7574355: Status 404 returned error can't find the container with id e3cd50ebe5edd441ac4051340d6b82bb8e23154f91a44a7199a61a86a7574355 Jul 31 20:31:22.451052 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.450968 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-fx8pm" event={"ID":"836b7c03-2d71-4d64-a018-a2f319e82592","Type":"ContainerStarted","Data":"9d2794ae2dd994d4f9849cc2fa9af17c8946a21494fbec6ed9ca2f1e3106cecb"} Jul 31 20:31:22.452781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.452742 2544 generic.go:356] "Generic (PLEG): container finished" podID="3a4c2b8b-926d-416c-bbc3-5af77737ed39" containerID="956fe2804a10640bf7b310cdbeee3456e034c11fa2a2479c471bd21d97e1ad50" exitCode=0 Jul 31 20:31:22.452937 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.452829 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh2qj" event={"ID":"3a4c2b8b-926d-416c-bbc3-5af77737ed39","Type":"ContainerDied","Data":"956fe2804a10640bf7b310cdbeee3456e034c11fa2a2479c471bd21d97e1ad50"} Jul 31 20:31:22.455781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.455751 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" event={"ID":"3dce5282-17a5-497e-a8dc-15e9c8cb8800","Type":"ContainerStarted","Data":"91a1c3086f28640f24bfb575130e83be5e903e7df64df4758981811aea977284"} Jul 31 20:31:22.457150 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.457120 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"e3cd50ebe5edd441ac4051340d6b82bb8e23154f91a44a7199a61a86a7574355"} Jul 31 20:31:22.459330 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.459298 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" event={"ID":"a9a01394-3275-4296-baee-b1ac06a37452","Type":"ContainerStarted","Data":"0db41342ff1eaf3a0d39cb2be3349f22103312c8f8547bd233f8b7ea124f7aa7"} Jul 31 20:31:22.459414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.459356 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" event={"ID":"a9a01394-3275-4296-baee-b1ac06a37452","Type":"ContainerStarted","Data":"a781877bdde05f0430120fdbbe18e9d06088a38ff5a51e2b83140bfe62d9675c"} Jul 31 20:31:22.459414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.459384 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" event={"ID":"a9a01394-3275-4296-baee-b1ac06a37452","Type":"ContainerStarted","Data":"26bd6034653df1e6aa144805ede7ab6e63e3f247b4aa25bb1dc61618d1ac6c54"} Jul 31 20:31:22.460772 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.460744 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"09a36766f300559e72ce979257987b6279cb09fd389ca389da16bdc6297aa879"} Jul 31 20:31:22.485654 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.485593 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-knvzx" podStartSLOduration=2.683436771 podStartE2EDuration="4.485569072s" totalImagesPullingTime="1.802132301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:22.484540036 +0000 UTC m=+60.860913216" watchObservedRunningTime="2026-07-31 20:31:22.485569072 +0000 UTC m=+60.861942252" Jul 31 20:31:22.503822 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.503490 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-hhprh" podStartSLOduration=1.9771865659999999 podStartE2EDuration="4.503468422s" totalImagesPullingTime="2.526281856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:22.503196088 +0000 UTC m=+60.879569268" watchObservedRunningTime="2026-07-31 20:31:22.503468422 +0000 UTC m=+60.879841603" Jul 31 20:31:22.883084 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.883047 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d8986d654-f46p4"] Jul 31 20:31:22.886025 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.885998 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.890342 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.890107 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 31 20:31:22.890650 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.890624 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fvahhd0282h2r\"" Jul 31 20:31:22.890849 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.890833 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 31 20:31:22.891169 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.891151 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 31 20:31:22.891373 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.891357 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 31 20:31:22.891568 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.891553 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w4qlj\"" Jul 31 20:31:22.897603 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.896239 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8986d654-f46p4"] Jul 31 20:31:22.956169 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956129 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-metrics-server-audit-profiles\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956355 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956187 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af20a449-b019-4665-8709-039623fefc2b-audit-log\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956355 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956221 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-tls\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956355 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956250 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwpq\" (UniqueName: \"kubernetes.io/projected/af20a449-b019-4665-8709-039623fefc2b-kube-api-access-mjwpq\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956355 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956300 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-client-certs\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956355 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956335 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:22.956630 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:22.956440 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-client-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057131 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057093 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-client-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057295 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057146 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-metrics-server-audit-profiles\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057745 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057720 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af20a449-b019-4665-8709-039623fefc2b-audit-log\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057777 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-tls\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057827 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057811 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwpq\" (UniqueName: \"kubernetes.io/projected/af20a449-b019-4665-8709-039623fefc2b-kube-api-access-mjwpq\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057971 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057890 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-client-certs\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.057971 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.057931 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.058192 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.058169 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-metrics-server-audit-profiles\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.058272 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.058222 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af20a449-b019-4665-8709-039623fefc2b-audit-log\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.058918 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.058885 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af20a449-b019-4665-8709-039623fefc2b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.060691 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.060665 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-client-ca-bundle\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.061293 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.061267 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-client-certs\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.062410 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.062384 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af20a449-b019-4665-8709-039623fefc2b-secret-metrics-server-tls\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.068317 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.068268 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwpq\" (UniqueName: \"kubernetes.io/projected/af20a449-b019-4665-8709-039623fefc2b-kube-api-access-mjwpq\") pod \"metrics-server-7d8986d654-f46p4\" (UID: \"af20a449-b019-4665-8709-039623fefc2b\") " pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.208178 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.204839 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:23.253379 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.252148 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2"] Jul 31 20:31:23.255271 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.254808 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.257720 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.257689 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5bctg\"" Jul 31 20:31:23.258097 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.257943 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 31 20:31:23.264291 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.264231 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2"] Jul 31 20:31:23.339441 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.339193 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-676fq" Jul 31 20:31:23.362513 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.361982 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.362513 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.362189 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zvz\" (UniqueName: \"kubernetes.io/projected/be14b32c-3b96-4155-ae28-b1237fec6e63-kube-api-access-b8zvz\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.393634 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.393594 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8986d654-f46p4"] Jul 31 20:31:23.396786 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:23.396752 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf20a449_b019_4665_8709_039623fefc2b.slice/crio-c6787cd253799e52852021482aef1b70de628b772e92fd5b12859d91e668dbe6 WatchSource:0}: Error finding container c6787cd253799e52852021482aef1b70de628b772e92fd5b12859d91e668dbe6: Status 404 returned error can't find the container with id c6787cd253799e52852021482aef1b70de628b772e92fd5b12859d91e668dbe6 Jul 31 20:31:23.463195 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.463146 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.463379 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.463260 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zvz\" (UniqueName: \"kubernetes.io/projected/be14b32c-3b96-4155-ae28-b1237fec6e63-kube-api-access-b8zvz\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.463445 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:23.463422 2544 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 31 20:31:23.463497 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:31:23.463474 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert podName:be14b32c-3b96-4155-ae28-b1237fec6e63 nodeName:}" failed. No retries permitted until 2026-07-31 20:31:23.963458447 +0000 UTC m=+62.339831604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-7rtj2" (UID: "be14b32c-3b96-4155-ae28-b1237fec6e63") : secret "monitoring-plugin-cert" not found Jul 31 20:31:23.466666 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.466572 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" event={"ID":"af20a449-b019-4665-8709-039623fefc2b","Type":"ContainerStarted","Data":"c6787cd253799e52852021482aef1b70de628b772e92fd5b12859d91e668dbe6"} Jul 31 20:31:23.471066 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.471033 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh2qj" event={"ID":"3a4c2b8b-926d-416c-bbc3-5af77737ed39","Type":"ContainerStarted","Data":"c4ae643e422b1e642597824f0f395365b0827fd80ac2f3a2f172708b282dca61"} Jul 31 20:31:23.471221 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.471094 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh2qj" event={"ID":"3a4c2b8b-926d-416c-bbc3-5af77737ed39","Type":"ContainerStarted","Data":"68c07fcc16b49168b81425591638a95bc8d657c230ef37c43d074f3ab9f51b5f"} Jul 31 20:31:23.475315 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.475288 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zvz\" (UniqueName: \"kubernetes.io/projected/be14b32c-3b96-4155-ae28-b1237fec6e63-kube-api-access-b8zvz\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.475473 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.475440 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd" exitCode=0 Jul 31 20:31:23.475798 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.475777 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd"} Jul 31 20:31:23.492593 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.492535 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fh2qj" podStartSLOduration=3.328384066 podStartE2EDuration="5.492517497s" totalImagesPullingTime="2.041861308s" totalInitContainerRuntime="122.272123ms" isStatefulPod=true podCreationTimestamp="2026-07-31 20:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:23.49163637 +0000 UTC m=+61.868009548" watchObservedRunningTime="2026-07-31 20:31:23.492517497 +0000 UTC m=+61.868890687" Jul 31 20:31:23.968852 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.968811 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:23.971553 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:23.971529 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be14b32c-3b96-4155-ae28-b1237fec6e63-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-7rtj2\" (UID: \"be14b32c-3b96-4155-ae28-b1237fec6e63\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:24.194781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.194745 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:24.360478 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.360043 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2"] Jul 31 20:31:24.363382 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:24.363347 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe14b32c_3b96_4155_ae28_b1237fec6e63.slice/crio-04404007dd6595b9ba06bacfb8f5e2d8f9d15d315b362ffcfd3e873e5afaa2e9 WatchSource:0}: Error finding container 04404007dd6595b9ba06bacfb8f5e2d8f9d15d315b362ffcfd3e873e5afaa2e9: Status 404 returned error can't find the container with id 04404007dd6595b9ba06bacfb8f5e2d8f9d15d315b362ffcfd3e873e5afaa2e9 Jul 31 20:31:24.483449 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.483280 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" event={"ID":"be14b32c-3b96-4155-ae28-b1237fec6e63","Type":"ContainerStarted","Data":"04404007dd6595b9ba06bacfb8f5e2d8f9d15d315b362ffcfd3e873e5afaa2e9"} Jul 31 20:31:24.487062 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.486910 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"22989bae29663028f5dc918a613eb50cab87b978cd9b4f44773bdcd10f606203"} Jul 31 20:31:24.487062 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.486991 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"c2f1c11e69bcdb2639e12dce8d5b7ac6adbe80b27a416f1877c76853a1d6000b"} Jul 31 20:31:24.487062 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.487019 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"bd02392bf382f48d8cd4d97414196cb403d6902a818fd88dbfc5e57a29b53b3e"} Jul 31 20:31:24.795675 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.795532 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:31:24.798902 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.798804 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.804468 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.804440 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 20:31:24.804662 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.804646 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5fmgm\"" Jul 31 20:31:24.804930 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.804915 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 20:31:24.805117 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.805101 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 20:31:24.805417 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.805401 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 20:31:24.805625 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.805609 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a7jrb9lmmvjt4\"" Jul 31 20:31:24.805783 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.805769 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 20:31:24.805984 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.805968 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 20:31:24.807055 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.806365 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 20:31:24.807055 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.806461 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 20:31:24.807055 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.806665 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 20:31:24.818065 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.817708 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 20:31:24.824358 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.823465 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 20:31:24.824358 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.824079 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 20:31:24.828300 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.828274 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 20:31:24.828725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.828697 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:31:24.837286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.832795 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880271 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgkw\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880326 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880370 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880419 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880456 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880483 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880511 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880538 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880569 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880593 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880636 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880667 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880695 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880729 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880771 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881146 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880791 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880806 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880850 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880895 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.881899 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.880924 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982018 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgkw\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982091 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982141 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982176 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982194 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982213 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982231 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982249 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982278 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982297 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982325 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982346 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982362 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982388 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982416 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982431 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.983224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982472 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.984363 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982542 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.984363 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982588 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.984363 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.982613 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.985204 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.985172 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.985998 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.985971 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.986829 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.986800 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.990535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.989459 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.990535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.989893 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.990535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.990139 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.990535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.990390 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.991900 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.990894 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.991900 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.991668 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.992999 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.992429 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.993854 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.993712 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.995152 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.995046 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.995608 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.995566 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.996206 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.995770 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.996206 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.996138 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.996206 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.996141 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.996721 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.996698 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:24.997171 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:24.997152 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:25.002230 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:25.002205 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:25.004351 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:25.004290 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgkw\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw\") pod \"prometheus-k8s-0\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:25.139496 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:25.139214 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:31:26.831250 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:26.831223 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:31:26.835115 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:26.835073 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb185a894_df6b_4352_869b_e1c3badb0a8e.slice/crio-44cc8d07c6a8bd40f8b4de6e6b5e26bcc5e54d43a1a248adf723dd7ad3275d2e WatchSource:0}: Error finding container 44cc8d07c6a8bd40f8b4de6e6b5e26bcc5e54d43a1a248adf723dd7ad3275d2e: Status 404 returned error can't find the container with id 44cc8d07c6a8bd40f8b4de6e6b5e26bcc5e54d43a1a248adf723dd7ad3275d2e Jul 31 20:31:27.503206 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.503145 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" event={"ID":"af20a449-b019-4665-8709-039623fefc2b","Type":"ContainerStarted","Data":"e451bc3d4b7eadd26934503b7177391f42b3c5d1377b9a01ae4ea9a0e34c9024"} Jul 31 20:31:27.507334 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507304 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098"} Jul 31 20:31:27.507492 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507386 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef"} Jul 31 20:31:27.507492 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507429 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1"} Jul 31 20:31:27.507492 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507468 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe"} Jul 31 20:31:27.507691 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507512 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a"} Jul 31 20:31:27.507691 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.507552 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerStarted","Data":"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493"} Jul 31 20:31:27.509065 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.509034 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" event={"ID":"be14b32c-3b96-4155-ae28-b1237fec6e63","Type":"ContainerStarted","Data":"7e39f8cd1fd9a95ad99e6350117b0956367dd79aa175a7f5bb0d9e1c1fe5aa3e"} Jul 31 20:31:27.509227 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.509211 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:27.511060 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.511034 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231" exitCode=0 Jul 31 20:31:27.511716 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.511694 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231"} Jul 31 20:31:27.511830 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.511789 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"44cc8d07c6a8bd40f8b4de6e6b5e26bcc5e54d43a1a248adf723dd7ad3275d2e"} Jul 31 20:31:27.515410 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.515388 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" Jul 31 20:31:27.516098 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.516075 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"9c6a8ca42cac0af9323773eb04190a4ed1f711c812a8caf86765552adf8624f2"} Jul 31 20:31:27.516187 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.516132 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"e8c8426df2cad8b8ab736793d0c4a94ab496f20dba194bdbfeb726056745f7ac"} Jul 31 20:31:27.516187 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.516159 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" event={"ID":"1f62f997-53ea-474f-beaa-5244f0d1470d","Type":"ContainerStarted","Data":"f5844569214892907e0d79f5affe7abd36694a54f313855b05711028c8ea514a"} Jul 31 20:31:27.516308 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.516267 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:27.520014 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.519966 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" podStartSLOduration=2.310395582 podStartE2EDuration="5.519947692s" totalImagesPullingTime="3.20955211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:27.518319835 +0000 UTC m=+65.894693015" watchObservedRunningTime="2026-07-31 20:31:27.519947692 +0000 UTC m=+65.896320872" Jul 31 20:31:27.539706 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.539647 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" podStartSLOduration=3.022645618 podStartE2EDuration="7.539629421s" totalImagesPullingTime="4.516983803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:27.538691596 +0000 UTC m=+65.915064802" watchObservedRunningTime="2026-07-31 20:31:27.539629421 +0000 UTC m=+65.916002591" Jul 31 20:31:27.554222 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.554156 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-7rtj2" podStartSLOduration=2.309739604 podStartE2EDuration="4.554133905s" totalImagesPullingTime="2.244394301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:27.552672952 +0000 UTC m=+65.929046132" watchObservedRunningTime="2026-07-31 20:31:27.554133905 +0000 UTC m=+65.930507088" Jul 31 20:31:27.603316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.603256 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.168809779 podStartE2EDuration="8.6032339s" totalImagesPullingTime="4.409068874s" totalInitContainerRuntime="25.355247ms" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:27.601365326 +0000 UTC m=+65.977738508" watchObservedRunningTime="2026-07-31 20:31:27.6032339 +0000 UTC m=+65.979607079" Jul 31 20:31:27.919993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.919961 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:27.919993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.919998 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:27.922663 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.922636 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 20:31:27.922798 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.922635 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 20:31:27.933298 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.933268 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99e6c527-9518-4a36-81c2-ea2634d4ecca-original-pull-secret\") pod \"global-pull-secret-syncer-b6gsd\" (UID: \"99e6c527-9518-4a36-81c2-ea2634d4ecca\") " pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:27.933553 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:27.933519 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad42f23-7c17-4dd6-abdf-47febed6fa7b-metrics-certs\") pod \"network-metrics-daemon-285jk\" (UID: \"8ad42f23-7c17-4dd6-abdf-47febed6fa7b\") " pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:28.021316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.021274 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:28.021632 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.021612 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9lfhf\"" Jul 31 20:31:28.023678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.023653 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 20:31:28.026606 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.026579 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6gsd" Jul 31 20:31:28.029703 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.029680 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-285jk" Jul 31 20:31:28.034007 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.033982 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 20:31:28.045964 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.045939 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gxp\" (UniqueName: \"kubernetes.io/projected/049da32c-25fd-41a3-9e0d-fd9ee564d56b-kube-api-access-v9gxp\") pod \"network-check-target-nm599\" (UID: \"049da32c-25fd-41a3-9e0d-fd9ee564d56b\") " pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:28.208360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.208324 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6gsd"] Jul 31 20:31:28.213380 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:28.213350 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e6c527_9518_4a36_81c2_ea2634d4ecca.slice/crio-4f4604829fbacec9d9d247a05b253114ea3224059f44fc6b061b5d8a3d2698b7 WatchSource:0}: Error finding container 4f4604829fbacec9d9d247a05b253114ea3224059f44fc6b061b5d8a3d2698b7: Status 404 returned error can't find the container with id 4f4604829fbacec9d9d247a05b253114ea3224059f44fc6b061b5d8a3d2698b7 Jul 31 20:31:28.231646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.231615 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-285jk"] Jul 31 20:31:28.237159 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:28.237113 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad42f23_7c17_4dd6_abdf_47febed6fa7b.slice/crio-dab13e9a3b641bfd7ae3cc80d1bb1dba2c5604e60a4b893f081bcdf1b7a641fc WatchSource:0}: Error finding container dab13e9a3b641bfd7ae3cc80d1bb1dba2c5604e60a4b893f081bcdf1b7a641fc: Status 404 returned error can't find the container with id dab13e9a3b641bfd7ae3cc80d1bb1dba2c5604e60a4b893f081bcdf1b7a641fc Jul 31 20:31:28.335344 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.335310 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6hmj\"" Jul 31 20:31:28.343346 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.343310 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:28.523619 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.523579 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6gsd" event={"ID":"99e6c527-9518-4a36-81c2-ea2634d4ecca","Type":"ContainerStarted","Data":"4f4604829fbacec9d9d247a05b253114ea3224059f44fc6b061b5d8a3d2698b7"} Jul 31 20:31:28.526699 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.526655 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-285jk" event={"ID":"8ad42f23-7c17-4dd6-abdf-47febed6fa7b","Type":"ContainerStarted","Data":"dab13e9a3b641bfd7ae3cc80d1bb1dba2c5604e60a4b893f081bcdf1b7a641fc"} Jul 31 20:31:28.535154 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:28.535125 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nm599"] Jul 31 20:31:28.539362 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:31:28.539327 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049da32c_25fd_41a3_9e0d_fd9ee564d56b.slice/crio-4b3cad9c6d3d46a982d6306640660391ce309f46923513753b8ae03e0fe91011 WatchSource:0}: Error finding container 4b3cad9c6d3d46a982d6306640660391ce309f46923513753b8ae03e0fe91011: Status 404 returned error can't find the container with id 4b3cad9c6d3d46a982d6306640660391ce309f46923513753b8ae03e0fe91011 Jul 31 20:31:29.535831 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:29.535744 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nm599" event={"ID":"049da32c-25fd-41a3-9e0d-fd9ee564d56b","Type":"ContainerStarted","Data":"4b3cad9c6d3d46a982d6306640660391ce309f46923513753b8ae03e0fe91011"} Jul 31 20:31:30.548734 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:30.547925 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-285jk" event={"ID":"8ad42f23-7c17-4dd6-abdf-47febed6fa7b","Type":"ContainerStarted","Data":"3499fea1f53edfdfa23d24dd481c6a3d27029d1a3c28c9e9f8266685eb33722a"} Jul 31 20:31:30.548734 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:30.547999 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-285jk" event={"ID":"8ad42f23-7c17-4dd6-abdf-47febed6fa7b","Type":"ContainerStarted","Data":"98bb37b60ce3e9e6fc37bc2e16ec499e683f72ceaae3874c40845a34deaec18e"} Jul 31 20:31:30.566860 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:30.566792 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-285jk" podStartSLOduration=66.899667562 podStartE2EDuration="1m8.566768268s" totalImagesPullingTime="1.667100706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:30.564408889 +0000 UTC m=+68.940782068" watchObservedRunningTime="2026-07-31 20:31:30.566768268 +0000 UTC m=+68.943141449" Jul 31 20:31:33.535590 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:33.535560 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f5d9b6d48-4kfg7" Jul 31 20:31:42.613369 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.613336 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825"} Jul 31 20:31:42.613806 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.613442 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3"} Jul 31 20:31:42.615469 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.615250 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6gsd" event={"ID":"99e6c527-9518-4a36-81c2-ea2634d4ecca","Type":"ContainerStarted","Data":"cfba56f8036297dc59d3c9f3b9ac5d4581f999360c0905e0da1cec5306422128"} Jul 31 20:31:42.616965 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.616943 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nm599" event={"ID":"049da32c-25fd-41a3-9e0d-fd9ee564d56b","Type":"ContainerStarted","Data":"f8bb111b174c89a84389993f2f17c9b68ef88991dfd78e3744b2cf0013464b86"} Jul 31 20:31:42.617091 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.617070 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:31:42.631993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.631946 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b6gsd" podStartSLOduration=66.681381633 podStartE2EDuration="1m20.631930853s" totalImagesPullingTime="13.95054922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:42.631141591 +0000 UTC m=+81.007514771" watchObservedRunningTime="2026-07-31 20:31:42.631930853 +0000 UTC m=+81.008304032" Jul 31 20:31:42.648467 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:42.648420 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nm599" podStartSLOduration=67.038850552 podStartE2EDuration="1m20.64839967s" totalImagesPullingTime="13.609549118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:42.64735032 +0000 UTC m=+81.023723498" watchObservedRunningTime="2026-07-31 20:31:42.64839967 +0000 UTC m=+81.024772880" Jul 31 20:31:43.205665 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.205623 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:43.205854 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.205699 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:31:43.627848 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.627804 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24"} Jul 31 20:31:43.628366 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.627953 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f"} Jul 31 20:31:43.628366 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.628006 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd"} Jul 31 20:31:43.628366 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.628034 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerStarted","Data":"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0"} Jul 31 20:31:43.630159 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.630131 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-fx8pm" event={"ID":"836b7c03-2d71-4d64-a018-a2f319e82592","Type":"ContainerStarted","Data":"0c1cc49c648e575f6ad44180642657c7a4b104e6630146a7acd6a0f8bdd43319"} Jul 31 20:31:43.630936 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.630440 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:43.645222 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.645191 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-fx8pm" Jul 31 20:31:43.657436 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.657377 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.992831409 podStartE2EDuration="19.65735951s" totalImagesPullingTime="14.638090787s" totalInitContainerRuntime="26.437314ms" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:43.65446822 +0000 UTC m=+82.030841424" watchObservedRunningTime="2026-07-31 20:31:43.65735951 +0000 UTC m=+82.033732688" Jul 31 20:31:43.670519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:43.670461 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-fx8pm" podStartSLOduration=3.7657184409999997 podStartE2EDuration="24.670438428s" totalImagesPullingTime="20.904719987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:31:43.668384571 +0000 UTC m=+82.044757751" watchObservedRunningTime="2026-07-31 20:31:43.670438428 +0000 UTC m=+82.046811612" Jul 31 20:31:45.140400 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:31:45.140357 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:03.210735 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:03.210704 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:32:03.214769 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:03.214751 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d8986d654-f46p4" Jul 31 20:32:13.632950 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:13.632913 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nm599" Jul 31 20:32:25.141373 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:25.139756 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:25.206350 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:25.206321 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:25.837726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:25.837697 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:38.845364 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845323 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:38.845972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845642 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="alertmanager" containerID="cri-o://9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" gracePeriod=120 Jul 31 20:32:38.845972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845828 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-web" containerID="cri-o://f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" gracePeriod=120 Jul 31 20:32:38.845972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845852 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="config-reloader" containerID="cri-o://52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" gracePeriod=120 Jul 31 20:32:38.845972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845917 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy" containerID="cri-o://e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" gracePeriod=120 Jul 31 20:32:38.845972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.845823 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="prom-label-proxy" containerID="cri-o://7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" gracePeriod=120 Jul 31 20:32:38.846256 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:38.846222 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-metric" containerID="cri-o://1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" gracePeriod=120 Jul 31 20:32:39.884505 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884469 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" exitCode=0 Jul 31 20:32:39.884505 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884496 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" exitCode=0 Jul 31 20:32:39.884505 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884502 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" exitCode=0 Jul 31 20:32:39.884505 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884508 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" exitCode=0 Jul 31 20:32:39.884999 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884540 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098"} Jul 31 20:32:39.884999 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884633 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1"} Jul 31 20:32:39.884999 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884660 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a"} Jul 31 20:32:39.884999 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:39.884684 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493"} Jul 31 20:32:40.195000 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.194977 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.273518 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273488 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273518 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273523 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpsqj\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273550 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273574 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273607 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273643 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273678 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273718 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.273752 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273746 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.274139 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273778 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.274139 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273806 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.274139 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273835 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.274139 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.273916 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy\") pod \"945a3d3d-2b11-4357-8e8a-bad63aef0394\" (UID: \"945a3d3d-2b11-4357-8e8a-bad63aef0394\") " Jul 31 20:32:40.274574 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.274541 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:40.275838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.275561 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:32:40.275838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.275765 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:40.278213 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278181 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.278213 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278198 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj" (OuterVolumeSpecName: "kube-api-access-wpsqj") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "kube-api-access-wpsqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:32:40.278402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278184 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out" (OuterVolumeSpecName: "config-out") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:32:40.278402 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278331 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.278521 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278482 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.278658 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278636 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:32:40.278748 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278663 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.278798 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.278783 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume" (OuterVolumeSpecName: "config-volume") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.282275 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.282249 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.288574 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.288544 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config" (OuterVolumeSpecName: "web-config") pod "945a3d3d-2b11-4357-8e8a-bad63aef0394" (UID: "945a3d3d-2b11-4357-8e8a-bad63aef0394"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:40.375018 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.374989 2544 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-volume\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375018 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375013 2544 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-metrics-client-ca\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375018 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375023 2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-main-db\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375034 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375042 2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/945a3d3d-2b11-4357-8e8a-bad63aef0394-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375053 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375069 2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/945a3d3d-2b11-4357-8e8a-bad63aef0394-config-out\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375078 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsqj\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-kube-api-access-wpsqj\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375086 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-main-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375113 2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-web-config\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375121 2544 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-cluster-tls-config\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375131 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/945a3d3d-2b11-4357-8e8a-bad63aef0394-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.375233 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.375139 2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/945a3d3d-2b11-4357-8e8a-bad63aef0394-tls-assets\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:40.891555 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891521 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" exitCode=0 Jul 31 20:32:40.891555 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891549 2544 generic.go:356] "Generic (PLEG): container finished" podID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerID="f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" exitCode=0 Jul 31 20:32:40.892017 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891603 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.892017 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891617 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef"} Jul 31 20:32:40.892017 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891690 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe"} Jul 31 20:32:40.892017 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891720 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"945a3d3d-2b11-4357-8e8a-bad63aef0394","Type":"ContainerDied","Data":"e3cd50ebe5edd441ac4051340d6b82bb8e23154f91a44a7199a61a86a7574355"} Jul 31 20:32:40.892017 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.891759 2544 scope.go:117] "RemoveContainer" containerID="7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" Jul 31 20:32:40.900338 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.900318 2544 scope.go:117] "RemoveContainer" containerID="1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" Jul 31 20:32:40.907534 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.907517 2544 scope.go:117] "RemoveContainer" containerID="e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" Jul 31 20:32:40.915074 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.915040 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:40.919587 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.919560 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:40.919915 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.919901 2544 scope.go:117] "RemoveContainer" containerID="f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" Jul 31 20:32:40.927685 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.927658 2544 scope.go:117] "RemoveContainer" containerID="52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" Jul 31 20:32:40.935481 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.935465 2544 scope.go:117] "RemoveContainer" containerID="9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" Jul 31 20:32:40.942029 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942003 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:40.942304 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942290 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-web" Jul 31 20:32:40.942304 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942304 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-web" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942317 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="config-reloader" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942323 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="config-reloader" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942331 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="init-config-reloader" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942336 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="init-config-reloader" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942342 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942347 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942357 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="alertmanager" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942362 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="alertmanager" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942376 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-metric" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942381 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-metric" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942388 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="prom-label-proxy" Jul 31 20:32:40.942430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942392 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="prom-label-proxy" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942439 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="alertmanager" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942447 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-metric" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942454 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="config-reloader" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942462 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="prom-label-proxy" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942469 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy-web" Jul 31 20:32:40.943129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.942475 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" containerName="kube-rbac-proxy" Jul 31 20:32:40.944074 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.944052 2544 scope.go:117] "RemoveContainer" containerID="50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd" Jul 31 20:32:40.946713 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.946695 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.949353 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949331 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 20:32:40.949449 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949370 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 20:32:40.949449 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949394 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 20:32:40.949646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949629 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 20:32:40.949726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949678 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 20:32:40.949726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949689 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 20:32:40.949829 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949735 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6df5p\"" Jul 31 20:32:40.949829 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949750 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 20:32:40.949829 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.949756 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 20:32:40.954056 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954036 2544 scope.go:117] "RemoveContainer" containerID="7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" Jul 31 20:32:40.954423 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.954396 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098\": container with ID starting with 7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098 not found: ID does not exist" containerID="7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" Jul 31 20:32:40.954519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954436 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098"} err="failed to get container status \"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098\": rpc error: code = NotFound desc = could not find container \"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098\": container with ID starting with 7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098 not found: ID does not exist" Jul 31 20:32:40.954519 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954480 2544 scope.go:117] "RemoveContainer" containerID="1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" Jul 31 20:32:40.954652 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954634 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 20:32:40.954733 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.954708 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef\": container with ID starting with 1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef not found: ID does not exist" containerID="1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" Jul 31 20:32:40.954791 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954742 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef"} err="failed to get container status \"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef\": rpc error: code = NotFound desc = could not find container \"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef\": container with ID starting with 1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef not found: ID does not exist" Jul 31 20:32:40.954791 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.954763 2544 scope.go:117] "RemoveContainer" containerID="e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" Jul 31 20:32:40.955116 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.955094 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1\": container with ID starting with e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1 not found: ID does not exist" containerID="e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" Jul 31 20:32:40.955183 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955126 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1"} err="failed to get container status \"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1\": rpc error: code = NotFound desc = could not find container \"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1\": container with ID starting with e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1 not found: ID does not exist" Jul 31 20:32:40.955183 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955152 2544 scope.go:117] "RemoveContainer" containerID="f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" Jul 31 20:32:40.955569 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.955550 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe\": container with ID starting with f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe not found: ID does not exist" containerID="f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" Jul 31 20:32:40.955655 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955578 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe"} err="failed to get container status \"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe\": rpc error: code = NotFound desc = could not find container \"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe\": container with ID starting with f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe not found: ID does not exist" Jul 31 20:32:40.955655 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955599 2544 scope.go:117] "RemoveContainer" containerID="52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" Jul 31 20:32:40.956015 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.955857 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a\": container with ID starting with 52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a not found: ID does not exist" containerID="52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" Jul 31 20:32:40.956015 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955903 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a"} err="failed to get container status \"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a\": rpc error: code = NotFound desc = could not find container \"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a\": container with ID starting with 52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a not found: ID does not exist" Jul 31 20:32:40.956015 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.955919 2544 scope.go:117] "RemoveContainer" containerID="9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" Jul 31 20:32:40.956198 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.956181 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493\": container with ID starting with 9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493 not found: ID does not exist" containerID="9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" Jul 31 20:32:40.956255 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956207 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493"} err="failed to get container status \"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493\": rpc error: code = NotFound desc = could not find container \"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493\": container with ID starting with 9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493 not found: ID does not exist" Jul 31 20:32:40.956255 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956228 2544 scope.go:117] "RemoveContainer" containerID="50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd" Jul 31 20:32:40.956487 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:40.956469 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd\": container with ID starting with 50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd not found: ID does not exist" containerID="50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd" Jul 31 20:32:40.956526 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956493 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd"} err="failed to get container status \"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd\": rpc error: code = NotFound desc = could not find container \"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd\": container with ID starting with 50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd not found: ID does not exist" Jul 31 20:32:40.956526 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956509 2544 scope.go:117] "RemoveContainer" containerID="7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098" Jul 31 20:32:40.956791 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956765 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098"} err="failed to get container status \"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098\": rpc error: code = NotFound desc = could not find container \"7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098\": container with ID starting with 7dc6a71eeba463cfbe771dbd7227fc4acbc44c901b0967edea8b2996c88f9098 not found: ID does not exist" Jul 31 20:32:40.956791 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.956791 2544 scope.go:117] "RemoveContainer" containerID="1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef" Jul 31 20:32:40.957091 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957069 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef"} err="failed to get container status \"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef\": rpc error: code = NotFound desc = could not find container \"1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef\": container with ID starting with 1335cb0d63522dd77c0fef39b9d68047f77cdee9d80682646c8f20f9adc8aeef not found: ID does not exist" Jul 31 20:32:40.957158 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957092 2544 scope.go:117] "RemoveContainer" containerID="e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1" Jul 31 20:32:40.957307 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957291 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1"} err="failed to get container status \"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1\": rpc error: code = NotFound desc = could not find container \"e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1\": container with ID starting with e30db6f3d72f1cd49343af245981d5347241f4f689098af69b32da4bf035b6c1 not found: ID does not exist" Jul 31 20:32:40.957307 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957307 2544 scope.go:117] "RemoveContainer" containerID="f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe" Jul 31 20:32:40.957551 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957518 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe"} err="failed to get container status \"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe\": rpc error: code = NotFound desc = could not find container \"f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe\": container with ID starting with f347b14a514f4fde137d07a920f1116bc36b313d7b20b97a8895c1c451d266fe not found: ID does not exist" Jul 31 20:32:40.957551 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.957546 2544 scope.go:117] "RemoveContainer" containerID="52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a" Jul 31 20:32:40.958803 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.958603 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a"} err="failed to get container status \"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a\": rpc error: code = NotFound desc = could not find container \"52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a\": container with ID starting with 52d2031f5404e600d636fe6db4f7de4b9bd24d58b08c28876f678587c5e9d48a not found: ID does not exist" Jul 31 20:32:40.958803 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.958631 2544 scope.go:117] "RemoveContainer" containerID="9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493" Jul 31 20:32:40.960935 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.960906 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493"} err="failed to get container status \"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493\": rpc error: code = NotFound desc = could not find container \"9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493\": container with ID starting with 9a5b5ffffd39aa4fca13e170bd129313c4fefe026768bc18a0b53d4a7cfa6493 not found: ID does not exist" Jul 31 20:32:40.960935 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.960936 2544 scope.go:117] "RemoveContainer" containerID="50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd" Jul 31 20:32:40.961281 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.961241 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd"} err="failed to get container status \"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd\": rpc error: code = NotFound desc = could not find container \"50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd\": container with ID starting with 50cd9d5bd9f04b8492752ca61e2fe0083810379d5ec6e8d56b46694a23494dbd not found: ID does not exist" Jul 31 20:32:40.961644 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.961623 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:40.980753 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980725 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.980887 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980758 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.980887 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980780 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmhf\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-kube-api-access-mvmhf\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.980887 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980804 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980894 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-web-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.980944 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981022 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981056 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981090 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-out\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981116 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981148 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981183 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:40.981269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:40.981212 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.081784 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081744 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.081784 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081785 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmhf\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-kube-api-access-mvmhf\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081823 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081893 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-web-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081918 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081965 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.081993 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082028 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-out\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082072 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082055 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082093 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082119 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082150 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.082433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.082215 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.083231 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.083176 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.083231 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.083221 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.083756 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.083728 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d28cabc-587e-49e7-9a8d-1c92f246be85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.085018 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.084999 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-out\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.085430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.085400 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.085535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.085409 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-web-config\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.085535 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.085507 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.085960 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.085937 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.086056 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.085985 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.086188 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.086169 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.086431 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.086409 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d28cabc-587e-49e7-9a8d-1c92f246be85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.087300 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.087285 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.089783 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.089761 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmhf\" (UniqueName: \"kubernetes.io/projected/7d28cabc-587e-49e7-9a8d-1c92f246be85-kube-api-access-mvmhf\") pod \"alertmanager-main-0\" (UID: \"7d28cabc-587e-49e7-9a8d-1c92f246be85\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.258544 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.258451 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 20:32:41.410376 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.410344 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 20:32:41.412592 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:32:41.412567 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d28cabc_587e_49e7_9a8d_1c92f246be85.slice/crio-b199ebef0b219bad04c2521ab3a78931c4923f0dbf75a593615df86a77acf9a0 WatchSource:0}: Error finding container b199ebef0b219bad04c2521ab3a78931c4923f0dbf75a593615df86a77acf9a0: Status 404 returned error can't find the container with id b199ebef0b219bad04c2521ab3a78931c4923f0dbf75a593615df86a77acf9a0 Jul 31 20:32:41.897567 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.897530 2544 generic.go:356] "Generic (PLEG): container finished" podID="7d28cabc-587e-49e7-9a8d-1c92f246be85" containerID="8edf5a3d66320fadd9ddefe67b09e2b8a6aebe1a6cf6fbd5bf263fce71963be4" exitCode=0 Jul 31 20:32:41.897967 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.897614 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerDied","Data":"8edf5a3d66320fadd9ddefe67b09e2b8a6aebe1a6cf6fbd5bf263fce71963be4"} Jul 31 20:32:41.897967 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:41.897704 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"b199ebef0b219bad04c2521ab3a78931c4923f0dbf75a593615df86a77acf9a0"} Jul 31 20:32:42.112453 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.112424 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945a3d3d-2b11-4357-8e8a-bad63aef0394" path="/var/lib/kubelet/pods/945a3d3d-2b11-4357-8e8a-bad63aef0394/volumes" Jul 31 20:32:42.848598 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.848552 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7489846744-5jshv"] Jul 31 20:32:42.852074 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.852042 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.855261 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855233 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 31 20:32:42.855422 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855293 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2kp4v\"" Jul 31 20:32:42.855553 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855535 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 31 20:32:42.855640 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855587 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 31 20:32:42.855640 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855623 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 31 20:32:42.855640 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.855633 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 31 20:32:42.860940 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.860823 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 31 20:32:42.864802 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.864778 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7489846744-5jshv"] Jul 31 20:32:42.902990 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.902962 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903011 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-serving-certs-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903037 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-metrics-client-ca\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903105 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903160 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-federate-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903276 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903313 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6nc\" (UniqueName: \"kubernetes.io/projected/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-kube-api-access-tf6nc\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.903483 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.903338 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:42.906673 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.906639 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"73d3dbc309c83cc5dcfd800fb7078b8b5e9c6248008d9221e66e597dc0ebebd3"} Jul 31 20:32:42.906820 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.906780 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"3af1db2c6d117572f4c05a754f05a6f5bf7babe7176201c3e9f5d4e663b6747e"} Jul 31 20:32:42.906919 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.906851 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"1eb666d26b5b0a447039588f18f23e5781dcb8a42ac521ca5d75b8033af2d46c"} Jul 31 20:32:42.906968 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.906921 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"b04bf68ffe090b386c0d51c7b12637436ebf4220c974161eae14c01ed441e6b8"} Jul 31 20:32:42.906968 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.906961 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"f92054d2b44a624738060796e4a477845ffa645c40c204288614cfa36ecd9a4d"} Jul 31 20:32:42.907054 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.907015 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d28cabc-587e-49e7-9a8d-1c92f246be85","Type":"ContainerStarted","Data":"a54c42fe12b7105b5da14a61e6328f7fce5de14752d3ea2e777db216d16f21a6"} Jul 31 20:32:42.931561 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:42.931504 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.914954116 podStartE2EDuration="2.931483741s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.529625ms" isStatefulPod=false podCreationTimestamp="2026-07-31 20:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:32:42.929699849 +0000 UTC m=+141.306073059" watchObservedRunningTime="2026-07-31 20:32:42.931483741 +0000 UTC m=+141.307856920" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004007 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004049 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6nc\" (UniqueName: \"kubernetes.io/projected/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-kube-api-access-tf6nc\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004079 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004160 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004203 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-serving-certs-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004316 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004229 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-metrics-client-ca\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.004763 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.004399 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.005575 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.005264 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-metrics-client-ca\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.005575 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.005369 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-federate-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.005575 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.005428 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.005575 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.005486 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-serving-certs-ca-bundle\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.007089 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.007058 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.007331 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.007311 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-telemeter-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.007737 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.007714 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-secret-telemeter-client\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.007788 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.007768 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-federate-client-tls\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.012430 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.012392 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6nc\" (UniqueName: \"kubernetes.io/projected/ed94aae4-944b-41b3-ada6-9cc6a78ff4e1-kube-api-access-tf6nc\") pod \"telemeter-client-7489846744-5jshv\" (UID: \"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1\") " pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.154965 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.154852 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:43.155232 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155198 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="prometheus" containerID="cri-o://c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3" gracePeriod=600 Jul 31 20:32:43.155389 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155239 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="thanos-sidecar" containerID="cri-o://c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0" gracePeriod=600 Jul 31 20:32:43.155389 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155239 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy" containerID="cri-o://a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f" gracePeriod=600 Jul 31 20:32:43.155389 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155300 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24" gracePeriod=600 Jul 31 20:32:43.155389 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155252 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-web" containerID="cri-o://4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd" gracePeriod=600 Jul 31 20:32:43.155389 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.155262 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="config-reloader" containerID="cri-o://491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825" gracePeriod=600 Jul 31 20:32:43.165380 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.165355 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" Jul 31 20:32:43.336271 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.336247 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7489846744-5jshv"] Jul 31 20:32:43.338313 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:32:43.338287 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded94aae4_944b_41b3_ada6_9cc6a78ff4e1.slice/crio-f296fd4ae4efa1b7800bb53ff1b7bf8f036c7daf0b2f55aa882e13c9c3a71814 WatchSource:0}: Error finding container f296fd4ae4efa1b7800bb53ff1b7bf8f036c7daf0b2f55aa882e13c9c3a71814: Status 404 returned error can't find the container with id f296fd4ae4efa1b7800bb53ff1b7bf8f036c7daf0b2f55aa882e13c9c3a71814 Jul 31 20:32:43.913086 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.913036 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" event={"ID":"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1","Type":"ContainerStarted","Data":"f296fd4ae4efa1b7800bb53ff1b7bf8f036c7daf0b2f55aa882e13c9c3a71814"} Jul 31 20:32:43.916741 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916716 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24" exitCode=0 Jul 31 20:32:43.916741 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916737 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f" exitCode=0 Jul 31 20:32:43.916741 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916743 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0" exitCode=0 Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916748 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825" exitCode=0 Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916755 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3" exitCode=0 Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916791 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24"} Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916891 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f"} Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916935 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0"} Jul 31 20:32:43.917019 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.916979 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825"} Jul 31 20:32:43.917219 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:43.917022 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3"} Jul 31 20:32:44.425984 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.425956 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:44.519466 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519380 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wgkw\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519466 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519417 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519466 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519441 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519484 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519506 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519526 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519573 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519598 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519633 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519659 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.519731 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519686 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519765 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519817 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519851 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519924 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519944 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519976 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.519988 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520004 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520030 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520067 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520103 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b185a894-df6b-4352-869b-e1c3badb0a8e\" (UID: \"b185a894-df6b-4352-869b-e1c3badb0a8e\") " Jul 31 20:32:44.520651 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520361 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-metrics-client-ca\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.520651 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520378 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.520651 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.520383 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.521685 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.521654 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.522489 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.522462 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.522584 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.522551 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:32:44.522976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.522950 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.523163 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.523110 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 20:32:44.523694 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.523665 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.524393 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.524364 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:32:44.524810 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.524783 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525047 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.524897 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config" (OuterVolumeSpecName: "config") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525118 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525099 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525321 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525285 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw" (OuterVolumeSpecName: "kube-api-access-9wgkw") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "kube-api-access-9wgkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:32:44.525415 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525325 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525494 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525470 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out" (OuterVolumeSpecName: "config-out") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 20:32:44.525557 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525532 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525637 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525616 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.525839 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.525821 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.537045 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.537021 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config" (OuterVolumeSpecName: "web-config") pod "b185a894-df6b-4352-869b-e1c3badb0a8e" (UID: "b185a894-df6b-4352-869b-e1c3badb0a8e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 20:32:44.621788 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621754 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.621788 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621788 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621799 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgkw\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-kube-api-access-9wgkw\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621811 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621827 2544 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-config\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621863 2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b185a894-df6b-4352-869b-e1c3badb0a8e-tls-assets\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621888 2544 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621902 2544 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-grpc-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621916 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621931 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621942 2544 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-metrics-client-certs\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621950 2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-config-out\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621958 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621966 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621974 2544 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-secret-kube-rbac-proxy\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621982 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-db\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.621989 2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b185a894-df6b-4352-869b-e1c3badb0a8e-web-config\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.622027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.622000 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b185a894-df6b-4352-869b-e1c3badb0a8e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:32:44.925947 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.925905 2544 generic.go:356] "Generic (PLEG): container finished" podID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerID="4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd" exitCode=0 Jul 31 20:32:44.926340 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.926009 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:44.926340 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.926010 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd"} Jul 31 20:32:44.926340 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.926147 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b185a894-df6b-4352-869b-e1c3badb0a8e","Type":"ContainerDied","Data":"44cc8d07c6a8bd40f8b4de6e6b5e26bcc5e54d43a1a248adf723dd7ad3275d2e"} Jul 31 20:32:44.926340 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.926181 2544 scope.go:117] "RemoveContainer" containerID="65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24" Jul 31 20:32:44.952056 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.952014 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:44.956113 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.956077 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:44.986032 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.985989 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:44.986434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986416 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="prometheus" Jul 31 20:32:44.986434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986435 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="prometheus" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986446 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-web" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986451 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-web" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986459 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="init-config-reloader" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986465 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="init-config-reloader" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986473 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="thanos-sidecar" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986478 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="thanos-sidecar" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986484 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-thanos" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986489 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-thanos" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986497 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986504 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986529 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="config-reloader" Jul 31 20:32:44.986552 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986535 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="config-reloader" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986591 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="thanos-sidecar" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986599 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986607 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-thanos" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986614 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="config-reloader" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986621 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="kube-rbac-proxy-web" Jul 31 20:32:44.986976 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.986627 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" containerName="prometheus" Jul 31 20:32:44.990646 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.990622 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:44.994627 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.994602 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 20:32:44.994780 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.994731 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 20:32:44.995050 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995027 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 20:32:44.995160 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995081 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 20:32:44.995221 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995025 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 20:32:44.995304 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995283 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 20:32:44.995747 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995534 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 20:32:44.995747 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995596 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a7jrb9lmmvjt4\"" Jul 31 20:32:44.996008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995933 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 20:32:44.996008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995951 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5fmgm\"" Jul 31 20:32:44.996008 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.995975 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 20:32:44.996241 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.996224 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 20:32:44.996293 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.996266 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 20:32:44.996486 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.996468 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 20:32:44.998041 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:44.998023 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 20:32:45.000321 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.000299 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 20:32:45.004678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.004657 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:45.025106 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025068 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025265 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025114 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025265 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025192 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025265 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025236 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025265 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025260 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025450 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025297 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025450 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025324 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025450 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025373 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025450 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025398 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025450 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025418 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025476 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025529 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025573 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025614 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025656 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025684 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025725 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025706 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlqc\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-kube-api-access-kqlqc\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025890 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025728 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025890 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025783 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.025890 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.025805 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.126996 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.126957 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.126996 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127005 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127041 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127070 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127096 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlqc\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-kube-api-access-kqlqc\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127125 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127158 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127186 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127224 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127262 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127244 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127277 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127307 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127331 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127370 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127399 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127427 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127454 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127476 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127531 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.127616 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.127582 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.128504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.128219 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.128504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.128250 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.128504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.128290 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.128504 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.128375 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.129198 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.128909 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.129904 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.129498 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.130834 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.130518 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.130834 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.130581 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.130834 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.130683 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.131894 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.131552 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.132165 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.132119 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.133432 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.133383 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.133432 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.133394 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.133432 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.133394 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.134336 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.134310 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.134736 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.134709 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.135128 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.135102 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.135350 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.135327 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.135939 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.135916 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.137124 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.137102 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlqc\" (UniqueName: \"kubernetes.io/projected/9c10e4b4-6373-4ee6-8bbb-e3e817630ada-kube-api-access-kqlqc\") pod \"prometheus-k8s-0\" (UID: \"9c10e4b4-6373-4ee6-8bbb-e3e817630ada\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.210891 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.210846 2544 scope.go:117] "RemoveContainer" containerID="a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f" Jul 31 20:32:45.230161 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.230140 2544 scope.go:117] "RemoveContainer" containerID="4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd" Jul 31 20:32:45.237448 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.237429 2544 scope.go:117] "RemoveContainer" containerID="c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0" Jul 31 20:32:45.244972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.244951 2544 scope.go:117] "RemoveContainer" containerID="491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825" Jul 31 20:32:45.252911 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.252858 2544 scope.go:117] "RemoveContainer" containerID="c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3" Jul 31 20:32:45.266407 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.266388 2544 scope.go:117] "RemoveContainer" containerID="44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231" Jul 31 20:32:45.278686 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.278665 2544 scope.go:117] "RemoveContainer" containerID="65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24" Jul 31 20:32:45.279012 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.278992 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24\": container with ID starting with 65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24 not found: ID does not exist" containerID="65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24" Jul 31 20:32:45.279102 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279022 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24"} err="failed to get container status \"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24\": rpc error: code = NotFound desc = could not find container \"65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24\": container with ID starting with 65ce361654125447e284686bc7760434dc05a241bc996f77933047f58934ef24 not found: ID does not exist" Jul 31 20:32:45.279102 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279042 2544 scope.go:117] "RemoveContainer" containerID="a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f" Jul 31 20:32:45.279348 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.279321 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f\": container with ID starting with a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f not found: ID does not exist" containerID="a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f" Jul 31 20:32:45.279414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279357 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f"} err="failed to get container status \"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f\": rpc error: code = NotFound desc = could not find container \"a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f\": container with ID starting with a2b3439c2ea8aa475fa9e8427bf3838db856b5b947771b94f65745edfd1b536f not found: ID does not exist" Jul 31 20:32:45.279414 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279379 2544 scope.go:117] "RemoveContainer" containerID="4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd" Jul 31 20:32:45.279647 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.279629 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd\": container with ID starting with 4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd not found: ID does not exist" containerID="4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd" Jul 31 20:32:45.279698 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279650 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd"} err="failed to get container status \"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd\": rpc error: code = NotFound desc = could not find container \"4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd\": container with ID starting with 4d7771d58e80c6793e453f6d33532f6b6311478a6e5b490b38001d8f733f33fd not found: ID does not exist" Jul 31 20:32:45.279698 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279665 2544 scope.go:117] "RemoveContainer" containerID="c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0" Jul 31 20:32:45.279981 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.279966 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0\": container with ID starting with c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0 not found: ID does not exist" containerID="c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0" Jul 31 20:32:45.280040 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279984 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0"} err="failed to get container status \"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0\": rpc error: code = NotFound desc = could not find container \"c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0\": container with ID starting with c9ed5744d843850f1a9e27fb735d24bd3f3fe9351f3dcc08f7aaf30e29c7bca0 not found: ID does not exist" Jul 31 20:32:45.280040 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.279998 2544 scope.go:117] "RemoveContainer" containerID="491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825" Jul 31 20:32:45.280241 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.280221 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825\": container with ID starting with 491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825 not found: ID does not exist" containerID="491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825" Jul 31 20:32:45.280333 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.280243 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825"} err="failed to get container status \"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825\": rpc error: code = NotFound desc = could not find container \"491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825\": container with ID starting with 491cfcaac90f86b01b62d10246c491211af80029cbdf53eaf4f41faa214cc825 not found: ID does not exist" Jul 31 20:32:45.280333 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.280256 2544 scope.go:117] "RemoveContainer" containerID="c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3" Jul 31 20:32:45.280483 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.280469 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3\": container with ID starting with c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3 not found: ID does not exist" containerID="c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3" Jul 31 20:32:45.280524 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.280487 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3"} err="failed to get container status \"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3\": rpc error: code = NotFound desc = could not find container \"c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3\": container with ID starting with c5351a8171cdf6059aa5775af67199228364929d939402528c2a589175baf0b3 not found: ID does not exist" Jul 31 20:32:45.280524 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.280499 2544 scope.go:117] "RemoveContainer" containerID="44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231" Jul 31 20:32:45.280775 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:32:45.280750 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231\": container with ID starting with 44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231 not found: ID does not exist" containerID="44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231" Jul 31 20:32:45.280893 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.280782 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231"} err="failed to get container status \"44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231\": rpc error: code = NotFound desc = could not find container \"44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231\": container with ID starting with 44a61ddd00aaa466f18cca2749c5d603c8bd2efcec7396699c22ab78b2979231 not found: ID does not exist" Jul 31 20:32:45.304831 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.304757 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:32:45.464536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.464443 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 20:32:45.932288 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.932254 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" event={"ID":"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1","Type":"ContainerStarted","Data":"05cabba5ac160acbc6273501ac7e5ec2b4a891a6483634c1dbea2dde47a7841f"} Jul 31 20:32:45.932745 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.932321 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" event={"ID":"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1","Type":"ContainerStarted","Data":"08a20aca0e41d8b7e8c7b94b1f2d2567839b53cbf815efcd650285e9fc4b71fe"} Jul 31 20:32:45.932745 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.932346 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" event={"ID":"ed94aae4-944b-41b3-ada6-9cc6a78ff4e1","Type":"ContainerStarted","Data":"fec11b3c58254d36895bae416b32709e0a5f17340617125ab7bbd270808d5e33"} Jul 31 20:32:45.934539 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.934512 2544 generic.go:356] "Generic (PLEG): container finished" podID="9c10e4b4-6373-4ee6-8bbb-e3e817630ada" containerID="ee97eff8160e80207e3c369cd66003617b0f5611a7c54e6a7df9d2fc66c3c14f" exitCode=0 Jul 31 20:32:45.934672 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.934567 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerDied","Data":"ee97eff8160e80207e3c369cd66003617b0f5611a7c54e6a7df9d2fc66c3c14f"} Jul 31 20:32:45.934672 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.934638 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"cd3c09af9bd144899686635b40a75e63925315d789805f0c4810378ee6a8487c"} Jul 31 20:32:45.952070 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:45.952015 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7489846744-5jshv" podStartSLOduration=2.053683378 podStartE2EDuration="3.951992675s" totalImagesPullingTime="1.898309297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:32:45.950155208 +0000 UTC m=+144.326528401" watchObservedRunningTime="2026-07-31 20:32:45.951992675 +0000 UTC m=+144.328365855" Jul 31 20:32:46.120151 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.120110 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b185a894-df6b-4352-869b-e1c3badb0a8e" path="/var/lib/kubelet/pods/b185a894-df6b-4352-869b-e1c3badb0a8e/volumes" Jul 31 20:32:46.944147 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944112 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"7f397dc2cbb6b4bedf5a96cd89a9f4b8a591d4c25d29376a554684bda57b6083"} Jul 31 20:32:46.944536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944178 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"3401da4397574b99f5d533bca07530319d7feb179b406e50673946800c1709e8"} Jul 31 20:32:46.944536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944204 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"c2a7a03bfbc94c739dc0c6fc745ee97e9cff4a90723ec1ba31299918838f8837"} Jul 31 20:32:46.944536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944234 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"b5b34b6593329b697e5cc0748970883f408fed07d12c65a4fc83d4dcef6eae76"} Jul 31 20:32:46.944536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944260 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"69cb5a58a4eb0188a911d680ad4772ef4f5cee34bf0e91610f117476388e95e3"} Jul 31 20:32:46.944536 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.944285 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c10e4b4-6373-4ee6-8bbb-e3e817630ada","Type":"ContainerStarted","Data":"871527e056ea536a6d1ef3f6d835b8daeae5b10a14d5d17cf1e4932c4ee8ff93"} Jul 31 20:32:46.976291 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:46.976230 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.953734924 podStartE2EDuration="2.976208133s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.473209ms" isStatefulPod=false podCreationTimestamp="2026-07-31 20:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:32:46.974099035 +0000 UTC m=+145.350472206" watchObservedRunningTime="2026-07-31 20:32:46.976208133 +0000 UTC m=+145.352581316" Jul 31 20:32:50.305224 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:32:50.305166 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:33:45.305563 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:33:45.305525 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:33:45.321969 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:33:45.321941 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:33:46.199199 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:33:46.199170 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 20:35:19.075085 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.075048 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl"] Jul 31 20:35:19.078510 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.078488 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.081188 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.081165 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 20:35:19.082214 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.082191 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fv6lr\"" Jul 31 20:35:19.082337 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.082194 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 20:35:19.086705 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.086682 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl"] Jul 31 20:35:19.179835 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.179795 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwbf\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-kube-api-access-ndwbf\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.180074 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.179958 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.281127 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.281086 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.281326 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.281160 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwbf\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-kube-api-access-ndwbf\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.289101 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.289065 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.289241 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.289222 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwbf\" (UniqueName: \"kubernetes.io/projected/ee8eccd1-612f-4022-93c9-f51db361a2d8-kube-api-access-ndwbf\") pod \"cert-manager-cainjector-559cdd8c87-4xgtl\" (UID: \"ee8eccd1-612f-4022-93c9-f51db361a2d8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.406736 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.406690 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" Jul 31 20:35:19.550465 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.550434 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl"] Jul 31 20:35:19.553965 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:35:19.553930 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee8eccd1_612f_4022_93c9_f51db361a2d8.slice/crio-5621a9859e4ed6d6d1145f4ebbb725513264e4bd8ef04eb2e2aed9b3df99e445 WatchSource:0}: Error finding container 5621a9859e4ed6d6d1145f4ebbb725513264e4bd8ef04eb2e2aed9b3df99e445: Status 404 returned error can't find the container with id 5621a9859e4ed6d6d1145f4ebbb725513264e4bd8ef04eb2e2aed9b3df99e445 Jul 31 20:35:19.559901 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:19.559838 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" event={"ID":"ee8eccd1-612f-4022-93c9-f51db361a2d8","Type":"ContainerStarted","Data":"5621a9859e4ed6d6d1145f4ebbb725513264e4bd8ef04eb2e2aed9b3df99e445"} Jul 31 20:35:21.340167 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.340059 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njpg7"] Jul 31 20:35:21.344026 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.344001 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.346563 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.346533 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8mgt9\"" Jul 31 20:35:21.354793 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.354659 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njpg7"] Jul 31 20:35:21.503614 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.503575 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-bound-sa-token\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.503816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.503647 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qll\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-kube-api-access-76qll\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.604667 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.604625 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-bound-sa-token\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.604897 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.604699 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qll\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-kube-api-access-76qll\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.614998 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.614969 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-bound-sa-token\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.615129 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.615114 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qll\" (UniqueName: \"kubernetes.io/projected/6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f-kube-api-access-76qll\") pod \"cert-manager-55b4d954d5-njpg7\" (UID: \"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f\") " pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:21.657360 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:21.657302 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-njpg7" Jul 31 20:35:23.026890 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.026847 2544 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 20:35:23.139838 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.139669 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njpg7"] Jul 31 20:35:23.142532 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:35:23.142497 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ecb6b62_f1d9_45e4_b3c7_36ad75f1d06f.slice/crio-3e322cddcc93f31d026f8bfe6443dbfccf59364ffa3624a44f52b0882b36da8d WatchSource:0}: Error finding container 3e322cddcc93f31d026f8bfe6443dbfccf59364ffa3624a44f52b0882b36da8d: Status 404 returned error can't find the container with id 3e322cddcc93f31d026f8bfe6443dbfccf59364ffa3624a44f52b0882b36da8d Jul 31 20:35:23.588069 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.588029 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" event={"ID":"ee8eccd1-612f-4022-93c9-f51db361a2d8","Type":"ContainerStarted","Data":"cbe1a778e71a1fabeece6a1cc1d7e225d27b8f2c20d4d4dfb35e934f3fde2cf6"} Jul 31 20:35:23.589580 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.589559 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-njpg7" event={"ID":"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f","Type":"ContainerStarted","Data":"2e9e098731cecd57b85b9cea86ff2f46bf8ee0210e0fd31526eb604d0378a4a1"} Jul 31 20:35:23.589714 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.589586 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-njpg7" event={"ID":"6ecb6b62-f1d9-45e4-b3c7-36ad75f1d06f","Type":"ContainerStarted","Data":"3e322cddcc93f31d026f8bfe6443dbfccf59364ffa3624a44f52b0882b36da8d"} Jul 31 20:35:23.605015 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.604969 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4xgtl" podStartSLOduration=1.094560168 podStartE2EDuration="4.604952712s" totalImagesPullingTime="3.510392544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:35:23.602939867 +0000 UTC m=+301.979313046" watchObservedRunningTime="2026-07-31 20:35:23.604952712 +0000 UTC m=+301.981325891" Jul 31 20:35:23.619749 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:23.619700 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-njpg7" podStartSLOduration=2.6196822600000003 podStartE2EDuration="2.61968226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:35:23.618593384 +0000 UTC m=+301.994966564" watchObservedRunningTime="2026-07-31 20:35:23.61968226 +0000 UTC m=+301.996055440" Jul 31 20:35:54.788510 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.788472 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7"] Jul 31 20:35:54.792322 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.792290 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797489 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g6mj8\"" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797517 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797532 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797551 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797563 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 31 20:35:54.797678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.797563 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 20:35:54.805145 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.805116 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7"] Jul 31 20:35:54.895547 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.895505 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-metrics-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.895726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.895566 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.895726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.895640 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e97f1994-18ef-42b8-ba3c-65ad09b7d720-manager-config\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.895726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.895685 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphjr\" (UniqueName: \"kubernetes.io/projected/e97f1994-18ef-42b8-ba3c-65ad09b7d720-kube-api-access-gphjr\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.997148 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.997112 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.997277 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.997160 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e97f1994-18ef-42b8-ba3c-65ad09b7d720-manager-config\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.997277 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.997203 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphjr\" (UniqueName: \"kubernetes.io/projected/e97f1994-18ef-42b8-ba3c-65ad09b7d720-kube-api-access-gphjr\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.997277 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.997268 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-metrics-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.997892 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.997846 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e97f1994-18ef-42b8-ba3c-65ad09b7d720-manager-config\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.999908 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.999887 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-metrics-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:54.999956 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:54.999928 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e97f1994-18ef-42b8-ba3c-65ad09b7d720-cert\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:55.013203 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:55.013171 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphjr\" (UniqueName: \"kubernetes.io/projected/e97f1994-18ef-42b8-ba3c-65ad09b7d720-kube-api-access-gphjr\") pod \"lws-controller-manager-595ffc7687-79dp7\" (UID: \"e97f1994-18ef-42b8-ba3c-65ad09b7d720\") " pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:55.106585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:55.106552 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:55.249782 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:55.249745 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7"] Jul 31 20:35:55.253266 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:35:55.253234 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97f1994_18ef_42b8_ba3c_65ad09b7d720.slice/crio-58942e04b1259267efd7da8934822ef0122bf3a16b96853d41be91e9cd4a362d WatchSource:0}: Error finding container 58942e04b1259267efd7da8934822ef0122bf3a16b96853d41be91e9cd4a362d: Status 404 returned error can't find the container with id 58942e04b1259267efd7da8934822ef0122bf3a16b96853d41be91e9cd4a362d Jul 31 20:35:55.255259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:55.255242 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:35:55.725287 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:55.725250 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" event={"ID":"e97f1994-18ef-42b8-ba3c-65ad09b7d720","Type":"ContainerStarted","Data":"58942e04b1259267efd7da8934822ef0122bf3a16b96853d41be91e9cd4a362d"} Jul 31 20:35:58.741320 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:58.741234 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" event={"ID":"e97f1994-18ef-42b8-ba3c-65ad09b7d720","Type":"ContainerStarted","Data":"0220f66010d1c197a1c1e44b7d1d86935e384331ea23f3de0616522c1ef3939a"} Jul 31 20:35:58.741320 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:58.741302 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:35:58.759144 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:35:58.759093 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" podStartSLOduration=1.622958892 podStartE2EDuration="4.75907773s" totalImagesPullingTime="3.136118838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:35:58.75724943 +0000 UTC m=+337.133622609" watchObservedRunningTime="2026-07-31 20:35:58.75907773 +0000 UTC m=+337.135450908" Jul 31 20:36:08.011173 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.011133 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2"] Jul 31 20:36:08.013710 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.013610 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.016165 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.016131 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 20:36:08.016299 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.016134 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 20:36:08.016299 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.016176 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 20:36:08.016299 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.016203 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-66k6x\"" Jul 31 20:36:08.016505 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.016490 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 20:36:08.033274 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.033245 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2"] Jul 31 20:36:08.123511 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.123470 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.123511 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.123507 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-manifests\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.123732 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.123525 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-webhook-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.123732 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.123621 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-tmp\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.123732 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.123673 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42bv\" (UniqueName: \"kubernetes.io/projected/e0035325-c47c-4bbd-9a7c-17636046bf60-kube-api-access-q42bv\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.224893 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.224827 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42bv\" (UniqueName: \"kubernetes.io/projected/e0035325-c47c-4bbd-9a7c-17636046bf60-kube-api-access-q42bv\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.225059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.224913 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.225059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.224945 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-manifests\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.225059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.224969 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-webhook-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.225059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.225033 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-tmp\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.225408 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.225382 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-manifests\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.227584 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.227551 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0035325-c47c-4bbd-9a7c-17636046bf60-tmp\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.227719 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.227655 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.227719 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.227687 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0035325-c47c-4bbd-9a7c-17636046bf60-webhook-cert\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.232021 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.231997 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42bv\" (UniqueName: \"kubernetes.io/projected/e0035325-c47c-4bbd-9a7c-17636046bf60-kube-api-access-q42bv\") pod \"opendatahub-operator-controller-manager-786d856849-zw2k2\" (UID: \"e0035325-c47c-4bbd-9a7c-17636046bf60\") " pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.324934 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.324805 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:08.470474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.470408 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2"] Jul 31 20:36:08.473119 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:36:08.473077 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0035325_c47c_4bbd_9a7c_17636046bf60.slice/crio-08f412827320e75650e7f17db39d0785e71f83df03eaab11e17f62bcfcd54848 WatchSource:0}: Error finding container 08f412827320e75650e7f17db39d0785e71f83df03eaab11e17f62bcfcd54848: Status 404 returned error can't find the container with id 08f412827320e75650e7f17db39d0785e71f83df03eaab11e17f62bcfcd54848 Jul 31 20:36:08.788528 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:08.788491 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" event={"ID":"e0035325-c47c-4bbd-9a7c-17636046bf60","Type":"ContainerStarted","Data":"08f412827320e75650e7f17db39d0785e71f83df03eaab11e17f62bcfcd54848"} Jul 31 20:36:09.749036 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:09.748955 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-595ffc7687-79dp7" Jul 31 20:36:11.806739 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:11.806703 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" event={"ID":"e0035325-c47c-4bbd-9a7c-17636046bf60","Type":"ContainerStarted","Data":"df707343de3fb8f442648362e7ce41fa0de85a0520004c6dd84a98fe5ecbd4d6"} Jul 31 20:36:13.817615 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:13.817580 2544 generic.go:356] "Generic (PLEG): container finished" podID="e0035325-c47c-4bbd-9a7c-17636046bf60" containerID="df707343de3fb8f442648362e7ce41fa0de85a0520004c6dd84a98fe5ecbd4d6" exitCode=0 Jul 31 20:36:13.818140 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:13.817662 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" event={"ID":"e0035325-c47c-4bbd-9a7c-17636046bf60","Type":"ContainerDied","Data":"df707343de3fb8f442648362e7ce41fa0de85a0520004c6dd84a98fe5ecbd4d6"} Jul 31 20:36:15.829248 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:15.829206 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" event={"ID":"e0035325-c47c-4bbd-9a7c-17636046bf60","Type":"ContainerStarted","Data":"afd7d867d369a9596a1791593d6cd534a541a62a3e56cbeeaa81e4f4913310ab"} Jul 31 20:36:15.829647 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:15.829426 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:15.850398 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:15.850344 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" podStartSLOduration=3.47834944 podStartE2EDuration="8.850329553s" totalImagesPullingTime="4.28724412s" totalInitContainerRuntime="1.084735993s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:36:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:36:15.849210382 +0000 UTC m=+354.225583558" watchObservedRunningTime="2026-07-31 20:36:15.850329553 +0000 UTC m=+354.226702731" Jul 31 20:36:26.836381 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:26.836347 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-786d856849-zw2k2" Jul 31 20:36:28.913941 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.913904 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d469b876d-2v26d"] Jul 31 20:36:28.926142 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.926112 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:28.928929 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.928891 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 20:36:28.929076 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.928891 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 20:36:28.929076 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.928944 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d469b876d-2v26d"] Jul 31 20:36:28.929076 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.929042 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 20:36:28.929321 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.929268 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 20:36:28.930029 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:28.930008 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wq66x\"" Jul 31 20:36:29.017319 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.017288 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tmp\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.017496 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.017324 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2njx\" (UniqueName: \"kubernetes.io/projected/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-kube-api-access-h2njx\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.017496 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.017357 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.117985 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.117948 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tmp\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.117985 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.117989 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2njx\" (UniqueName: \"kubernetes.io/projected/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-kube-api-access-h2njx\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.118212 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.118020 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.118212 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:36:29.118125 2544 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 31 20:36:29.118212 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:36:29.118191 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs podName:8b7fad3e-521b-4699-bbf8-b8d5b29dabeb nodeName:}" failed. No retries permitted until 2026-07-31 20:36:29.618175081 +0000 UTC m=+367.994548237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs") pod "kube-auth-proxy-6d469b876d-2v26d" (UID: "8b7fad3e-521b-4699-bbf8-b8d5b29dabeb") : secret "kube-auth-proxy-tls" not found Jul 31 20:36:29.120453 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.120425 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tmp\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.136201 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.136176 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2njx\" (UniqueName: \"kubernetes.io/projected/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-kube-api-access-h2njx\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.623106 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.623067 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.625705 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.625686 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7fad3e-521b-4699-bbf8-b8d5b29dabeb-tls-certs\") pod \"kube-auth-proxy-6d469b876d-2v26d\" (UID: \"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb\") " pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.837694 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.837656 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" Jul 31 20:36:29.983344 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:29.983308 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d469b876d-2v26d"] Jul 31 20:36:29.986151 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:36:29.986123 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7fad3e_521b_4699_bbf8_b8d5b29dabeb.slice/crio-53bdd0fb5cee8a35f2a597707d641826f1b8a0592708e875e86e174997204745 WatchSource:0}: Error finding container 53bdd0fb5cee8a35f2a597707d641826f1b8a0592708e875e86e174997204745: Status 404 returned error can't find the container with id 53bdd0fb5cee8a35f2a597707d641826f1b8a0592708e875e86e174997204745 Jul 31 20:36:30.901503 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:30.901464 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" event={"ID":"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb","Type":"ContainerStarted","Data":"53bdd0fb5cee8a35f2a597707d641826f1b8a0592708e875e86e174997204745"} Jul 31 20:36:32.912537 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:32.912500 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" event={"ID":"8b7fad3e-521b-4699-bbf8-b8d5b29dabeb","Type":"ContainerStarted","Data":"f6dd9bd0233432f8222e727bfcbeb4605dabd53ec98d6dd425a32e0d68d4fe28"} Jul 31 20:36:32.929089 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:36:32.929030 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d469b876d-2v26d" podStartSLOduration=2.530378748 podStartE2EDuration="4.929010793s" totalImagesPullingTime="2.398632045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:36:32.92806883 +0000 UTC m=+371.304442009" watchObservedRunningTime="2026-07-31 20:36:32.929010793 +0000 UTC m=+371.305383973" Jul 31 20:37:58.063296 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.063215 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9"] Jul 31 20:37:58.066390 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.066373 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.068819 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.068796 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 31 20:37:58.068929 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.068822 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fbg8w\"" Jul 31 20:37:58.068929 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.068805 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 20:37:58.069878 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.069841 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 31 20:37:58.069972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.069856 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 20:37:58.073936 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.073910 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9"] Jul 31 20:37:58.117378 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.117349 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.117378 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.117382 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqht\" (UniqueName: \"kubernetes.io/projected/59aca7dd-abfd-47f7-8f52-d51f650b2617-kube-api-access-wqqht\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.117629 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.117481 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59aca7dd-abfd-47f7-8f52-d51f650b2617-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.218062 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.218023 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.218276 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.218080 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqht\" (UniqueName: \"kubernetes.io/projected/59aca7dd-abfd-47f7-8f52-d51f650b2617-kube-api-access-wqqht\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.218276 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.218171 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59aca7dd-abfd-47f7-8f52-d51f650b2617-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.218276 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:37:58.218190 2544 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 31 20:37:58.218440 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:37:58.218280 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert podName:59aca7dd-abfd-47f7-8f52-d51f650b2617 nodeName:}" failed. No retries permitted until 2026-07-31 20:37:58.71825736 +0000 UTC m=+457.094630531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-qpmg9" (UID: "59aca7dd-abfd-47f7-8f52-d51f650b2617") : secret "plugin-serving-cert" not found Jul 31 20:37:58.218854 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.218837 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/59aca7dd-abfd-47f7-8f52-d51f650b2617-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.228228 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.228197 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqht\" (UniqueName: \"kubernetes.io/projected/59aca7dd-abfd-47f7-8f52-d51f650b2617-kube-api-access-wqqht\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.723035 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.722996 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.725584 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.725562 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/59aca7dd-abfd-47f7-8f52-d51f650b2617-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qpmg9\" (UID: \"59aca7dd-abfd-47f7-8f52-d51f650b2617\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:58.976746 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:58.976649 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" Jul 31 20:37:59.110269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:59.110241 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9"] Jul 31 20:37:59.113013 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:37:59.112983 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59aca7dd_abfd_47f7_8f52_d51f650b2617.slice/crio-abe97a0fc585929917362f7527a113a226ebe5f0737f0f2c4a882de6912d0e81 WatchSource:0}: Error finding container abe97a0fc585929917362f7527a113a226ebe5f0737f0f2c4a882de6912d0e81: Status 404 returned error can't find the container with id abe97a0fc585929917362f7527a113a226ebe5f0737f0f2c4a882de6912d0e81 Jul 31 20:37:59.314318 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:37:59.314227 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" event={"ID":"59aca7dd-abfd-47f7-8f52-d51f650b2617","Type":"ContainerStarted","Data":"abe97a0fc585929917362f7527a113a226ebe5f0737f0f2c4a882de6912d0e81"} Jul 31 20:38:04.343974 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:04.343856 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" event={"ID":"59aca7dd-abfd-47f7-8f52-d51f650b2617","Type":"ContainerStarted","Data":"520bf4c8d619888cc30e8ee1619cee20d01289d6aa3a4fae3ff146acd2cf30d6"} Jul 31 20:38:04.362783 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:04.362730 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qpmg9" podStartSLOduration=1.534707973 podStartE2EDuration="6.362714247s" totalImagesPullingTime="4.828006274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:38:04.360011645 +0000 UTC m=+462.736384824" watchObservedRunningTime="2026-07-31 20:38:04.362714247 +0000 UTC m=+462.739087425" Jul 31 20:38:52.296293 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.296250 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:38:52.299804 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.299780 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.302094 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.302071 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 20:38:52.306375 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.306349 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:38:52.334334 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.334298 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:38:52.443971 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.443940 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66105734-628b-47f1-b73c-e1d420d3a6bb-config-file\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.443971 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.443989 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/66105734-628b-47f1-b73c-e1d420d3a6bb-kube-api-access-xzvsm\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.544781 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.544743 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66105734-628b-47f1-b73c-e1d420d3a6bb-config-file\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.545031 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.544808 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/66105734-628b-47f1-b73c-e1d420d3a6bb-kube-api-access-xzvsm\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.545397 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.545379 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66105734-628b-47f1-b73c-e1d420d3a6bb-config-file\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.554148 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.554081 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/66105734-628b-47f1-b73c-e1d420d3a6bb-kube-api-access-xzvsm\") pod \"limitador-limitador-85d9db4d5d-dx8cg\" (UID: \"66105734-628b-47f1-b73c-e1d420d3a6bb\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.611317 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.611278 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:52.747617 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.747586 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:38:52.750252 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:38:52.750222 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66105734_628b_47f1_b73c_e1d420d3a6bb.slice/crio-8ebac67645f49a06dd9e09f73988d7453daaf19c0fdb71406f7cef45a6749b6e WatchSource:0}: Error finding container 8ebac67645f49a06dd9e09f73988d7453daaf19c0fdb71406f7cef45a6749b6e: Status 404 returned error can't find the container with id 8ebac67645f49a06dd9e09f73988d7453daaf19c0fdb71406f7cef45a6749b6e Jul 31 20:38:52.862806 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.862772 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:38:52.867439 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.867420 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:38:52.870057 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.870034 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhqb4\"" Jul 31 20:38:52.872474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.872450 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:38:52.949286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:52.949245 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvl6\" (UniqueName: \"kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6\") pod \"authorino-c548b5d7d-njbqn\" (UID: \"ac37a61e-fb86-42fd-8e20-fd049aeec86a\") " pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:38:53.050585 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.050543 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvl6\" (UniqueName: \"kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6\") pod \"authorino-c548b5d7d-njbqn\" (UID: \"ac37a61e-fb86-42fd-8e20-fd049aeec86a\") " pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:38:53.059083 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.059049 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvl6\" (UniqueName: \"kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6\") pod \"authorino-c548b5d7d-njbqn\" (UID: \"ac37a61e-fb86-42fd-8e20-fd049aeec86a\") " pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:38:53.178156 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.178059 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:38:53.312812 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.312777 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:38:53.316268 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:38:53.316244 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac37a61e_fb86_42fd_8e20_fd049aeec86a.slice/crio-f0b79b630e8d1b16a8b1e727a9ea601efae8bb3ec45dcb06498fd459f67e72a1 WatchSource:0}: Error finding container f0b79b630e8d1b16a8b1e727a9ea601efae8bb3ec45dcb06498fd459f67e72a1: Status 404 returned error can't find the container with id f0b79b630e8d1b16a8b1e727a9ea601efae8bb3ec45dcb06498fd459f67e72a1 Jul 31 20:38:53.581820 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.581737 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" event={"ID":"66105734-628b-47f1-b73c-e1d420d3a6bb","Type":"ContainerStarted","Data":"8ebac67645f49a06dd9e09f73988d7453daaf19c0fdb71406f7cef45a6749b6e"} Jul 31 20:38:53.583190 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:53.583162 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-njbqn" event={"ID":"ac37a61e-fb86-42fd-8e20-fd049aeec86a","Type":"ContainerStarted","Data":"f0b79b630e8d1b16a8b1e727a9ea601efae8bb3ec45dcb06498fd459f67e72a1"} Jul 31 20:38:56.601995 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:56.601940 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" event={"ID":"66105734-628b-47f1-b73c-e1d420d3a6bb","Type":"ContainerStarted","Data":"43994591817c2c7200e5b769554696b0964331ae5bd294be928aa3ce48642eee"} Jul 31 20:38:56.602474 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:56.602030 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:38:56.603473 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:56.603449 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-njbqn" event={"ID":"ac37a61e-fb86-42fd-8e20-fd049aeec86a","Type":"ContainerStarted","Data":"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd"} Jul 31 20:38:56.617198 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:56.617148 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" podStartSLOduration=1.755854407 podStartE2EDuration="4.617130342s" totalImagesPullingTime="2.861275935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:38:56.616231944 +0000 UTC m=+514.992605124" watchObservedRunningTime="2026-07-31 20:38:56.617130342 +0000 UTC m=+514.993503574" Jul 31 20:38:56.630100 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:38:56.630040 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-njbqn" podStartSLOduration=2.327934957 podStartE2EDuration="4.630019987s" totalImagesPullingTime="2.30208503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:38:56.629219294 +0000 UTC m=+515.005592490" watchObservedRunningTime="2026-07-31 20:38:56.630019987 +0000 UTC m=+515.006393167" Jul 31 20:39:07.609336 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:07.609301 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg" Jul 31 20:39:24.410264 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.410224 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:39:24.410821 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.410420 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-njbqn" podUID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" containerName="authorino" containerID="cri-o://30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd" gracePeriod=30 Jul 31 20:39:24.665251 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.665185 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:39:24.727059 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.727021 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmvl6\" (UniqueName: \"kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6\") pod \"ac37a61e-fb86-42fd-8e20-fd049aeec86a\" (UID: \"ac37a61e-fb86-42fd-8e20-fd049aeec86a\") " Jul 31 20:39:24.729491 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.729459 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6" (OuterVolumeSpecName: "kube-api-access-tmvl6") pod "ac37a61e-fb86-42fd-8e20-fd049aeec86a" (UID: "ac37a61e-fb86-42fd-8e20-fd049aeec86a"). InnerVolumeSpecName "kube-api-access-tmvl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 20:39:24.739290 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.739259 2544 generic.go:356] "Generic (PLEG): container finished" podID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" containerID="30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd" exitCode=0 Jul 31 20:39:24.739394 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.739310 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-njbqn" Jul 31 20:39:24.739394 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.739340 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-njbqn" event={"ID":"ac37a61e-fb86-42fd-8e20-fd049aeec86a","Type":"ContainerDied","Data":"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd"} Jul 31 20:39:24.739394 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.739377 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-njbqn" event={"ID":"ac37a61e-fb86-42fd-8e20-fd049aeec86a","Type":"ContainerDied","Data":"f0b79b630e8d1b16a8b1e727a9ea601efae8bb3ec45dcb06498fd459f67e72a1"} Jul 31 20:39:24.739491 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.739396 2544 scope.go:117] "RemoveContainer" containerID="30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd" Jul 31 20:39:24.748963 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.748947 2544 scope.go:117] "RemoveContainer" containerID="30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd" Jul 31 20:39:24.749238 ip-10-0-141-177 kubenswrapper[2544]: E0731 20:39:24.749220 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd\": container with ID starting with 30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd not found: ID does not exist" containerID="30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd" Jul 31 20:39:24.749292 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.749249 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd"} err="failed to get container status \"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd\": rpc error: code = NotFound desc = could not find container \"30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd\": container with ID starting with 30452213e21acf00e4a1f6384398c625ffde99eed7bf7ff6c67963f4a306a7fd not found: ID does not exist" Jul 31 20:39:24.760423 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.760400 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:39:24.764189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.764161 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-njbqn"] Jul 31 20:39:24.828088 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:24.828050 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvl6\" (UniqueName: \"kubernetes.io/projected/ac37a61e-fb86-42fd-8e20-fd049aeec86a-kube-api-access-tmvl6\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 20:39:26.112964 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:39:26.112928 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" path="/var/lib/kubelet/pods/ac37a61e-fb86-42fd-8e20-fd049aeec86a/volumes" Jul 31 20:40:04.854448 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:04.854408 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:40:47.179927 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.179819 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v"] Jul 31 20:40:47.182259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.180287 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" containerName="authorino" Jul 31 20:40:47.182259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.180302 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" containerName="authorino" Jul 31 20:40:47.182259 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.180382 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac37a61e-fb86-42fd-8e20-fd049aeec86a" containerName="authorino" Jul 31 20:40:47.183077 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.183062 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.185682 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.185660 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6pmz\"" Jul 31 20:40:47.186590 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.186563 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 20:40:47.186897 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.186853 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 31 20:40:47.186993 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.186897 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 20:40:47.196478 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.196447 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v"] Jul 31 20:40:47.287027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.286993 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.287027 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.287032 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.287246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.287063 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkhn\" (UniqueName: \"kubernetes.io/projected/b2cedb60-722e-443b-80e5-9b98bde9f3b0-kube-api-access-nzkhn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.287246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.287110 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.287246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.287157 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.287246 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.287217 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388547 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388492 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388547 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388540 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388794 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388578 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkhn\" (UniqueName: \"kubernetes.io/projected/b2cedb60-722e-443b-80e5-9b98bde9f3b0-kube-api-access-nzkhn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388794 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388608 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388794 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388633 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.388794 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.388687 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.389042 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.389019 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.389094 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.389057 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.389094 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.389087 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.391189 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.391166 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cedb60-722e-443b-80e5-9b98bde9f3b0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.391424 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.391402 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cedb60-722e-443b-80e5-9b98bde9f3b0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.397363 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.397339 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkhn\" (UniqueName: \"kubernetes.io/projected/b2cedb60-722e-443b-80e5-9b98bde9f3b0-kube-api-access-nzkhn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v\" (UID: \"b2cedb60-722e-443b-80e5-9b98bde9f3b0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.503898 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.503791 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:47.581772 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.581736 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:40:47.645540 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:47.645506 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v"] Jul 31 20:40:47.648944 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:40:47.648917 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cedb60_722e_443b_80e5_9b98bde9f3b0.slice/crio-b9cb283754d31e74269de116a450ee429ebe4ea31517506b2e594f7cef08fcab WatchSource:0}: Error finding container b9cb283754d31e74269de116a450ee429ebe4ea31517506b2e594f7cef08fcab: Status 404 returned error can't find the container with id b9cb283754d31e74269de116a450ee429ebe4ea31517506b2e594f7cef08fcab Jul 31 20:40:48.148436 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:48.148394 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" event={"ID":"b2cedb60-722e-443b-80e5-9b98bde9f3b0","Type":"ContainerStarted","Data":"b9cb283754d31e74269de116a450ee429ebe4ea31517506b2e594f7cef08fcab"} Jul 31 20:40:49.154245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:49.154202 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" event={"ID":"b2cedb60-722e-443b-80e5-9b98bde9f3b0","Type":"ContainerStarted","Data":"0db1502dec8a17e78609e964a03ee0097af20730a1fc150c9cfa176405415927"} Jul 31 20:40:49.171679 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:49.171619 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" podStartSLOduration=0.993394812 podStartE2EDuration="2.171596896s" totalImagesPullingTime="1.178202084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:40:49.171248388 +0000 UTC m=+627.547621568" watchObservedRunningTime="2026-07-31 20:40:49.171596896 +0000 UTC m=+627.547970078" Jul 31 20:40:57.504163 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:57.504112 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:57.504163 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:57.504184 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:57.516766 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:57.516741 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:40:58.206658 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:40:58.206626 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v" Jul 31 20:41:05.283240 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:05.283200 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:41:18.884792 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:18.884749 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:41:35.581726 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.581688 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf"] Jul 31 20:41:35.588071 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.588047 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.590624 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.590604 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 31 20:41:35.594487 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.594456 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf"] Jul 31 20:41:35.628544 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628515 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.628722 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628547 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.628722 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628571 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.628722 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628667 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d205be-3a52-496d-af8d-6e238cc57448-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.628861 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628723 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.628861 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.628755 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5xw\" (UniqueName: \"kubernetes.io/projected/89d205be-3a52-496d-af8d-6e238cc57448-kube-api-access-4p5xw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.729862 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.729819 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d205be-3a52-496d-af8d-6e238cc57448-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.729862 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.729896 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730223 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.729926 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5xw\" (UniqueName: \"kubernetes.io/projected/89d205be-3a52-496d-af8d-6e238cc57448-kube-api-access-4p5xw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730223 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730001 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730223 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730023 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730223 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730055 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730464 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730441 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730515 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730486 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.730515 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.730497 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.732347 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.732327 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d205be-3a52-496d-af8d-6e238cc57448-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.732494 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.732476 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d205be-3a52-496d-af8d-6e238cc57448-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.737510 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.737491 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5xw\" (UniqueName: \"kubernetes.io/projected/89d205be-3a52-496d-af8d-6e238cc57448-kube-api-access-4p5xw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf\" (UID: \"89d205be-3a52-496d-af8d-6e238cc57448\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:35.906172 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:35.906131 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:36.045565 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.045537 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf"] Jul 31 20:41:36.048240 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:41:36.048212 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d205be_3a52_496d_af8d_6e238cc57448.slice/crio-da50ae31d236cc9eeab5a251d4f2c949ef51f2cb4af2ce4dcd6b352d77bab425 WatchSource:0}: Error finding container da50ae31d236cc9eeab5a251d4f2c949ef51f2cb4af2ce4dcd6b352d77bab425: Status 404 returned error can't find the container with id da50ae31d236cc9eeab5a251d4f2c949ef51f2cb4af2ce4dcd6b352d77bab425 Jul 31 20:41:36.049602 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.049583 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 20:41:36.384177 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.384142 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" event={"ID":"89d205be-3a52-496d-af8d-6e238cc57448","Type":"ContainerStarted","Data":"d2540cb49d6a43a341f9f81fc25e7ffa37ca018942a92fc26b478f05c736887c"} Jul 31 20:41:36.384353 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.384190 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" event={"ID":"89d205be-3a52-496d-af8d-6e238cc57448","Type":"ContainerStarted","Data":"da50ae31d236cc9eeab5a251d4f2c949ef51f2cb4af2ce4dcd6b352d77bab425"} Jul 31 20:41:36.402165 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.402114 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" podStartSLOduration=1.237289382 podStartE2EDuration="1.402098079s" totalImagesPullingTime="164.808697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:41:36.399948105 +0000 UTC m=+674.776321281" watchObservedRunningTime="2026-07-31 20:41:36.402098079 +0000 UTC m=+674.778471258" Jul 31 20:41:36.976286 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:36.976248 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:41:43.985445 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:43.985401 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:41:45.906816 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:45.906771 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:45.907291 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:45.906886 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:45.919569 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:45.919540 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:46.444245 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:46.444216 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf" Jul 31 20:41:51.382972 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:41:51.382923 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:42:47.233375 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.233333 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7564444965-4pjl8"] Jul 31 20:42:47.237479 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.237460 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.240911 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.240854 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 31 20:42:47.240911 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.240853 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhqb4\"" Jul 31 20:42:47.246331 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.246308 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7564444965-4pjl8"] Jul 31 20:42:47.410567 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.410528 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xgw\" (UniqueName: \"kubernetes.io/projected/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-kube-api-access-s8xgw\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.410757 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.410587 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-tls-cert\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.512142 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.512050 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xgw\" (UniqueName: \"kubernetes.io/projected/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-kube-api-access-s8xgw\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.512142 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.512119 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-tls-cert\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.514841 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.514814 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-tls-cert\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.520878 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.520831 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xgw\" (UniqueName: \"kubernetes.io/projected/6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98-kube-api-access-s8xgw\") pod \"authorino-7564444965-4pjl8\" (UID: \"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98\") " pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.547740 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.547685 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564444965-4pjl8" Jul 31 20:42:47.688498 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.688472 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7564444965-4pjl8"] Jul 31 20:42:47.691444 ip-10-0-141-177 kubenswrapper[2544]: W0731 20:42:47.691417 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dbf9971_3409_44d2_9e9a_cb8e2e2e3b98.slice/crio-2b83888e40e596804ac3f4bdcd3dc589634bec914b2bf8da5fe22c3591a4cb1f WatchSource:0}: Error finding container 2b83888e40e596804ac3f4bdcd3dc589634bec914b2bf8da5fe22c3591a4cb1f: Status 404 returned error can't find the container with id 2b83888e40e596804ac3f4bdcd3dc589634bec914b2bf8da5fe22c3591a4cb1f Jul 31 20:42:47.725433 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:47.725400 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564444965-4pjl8" event={"ID":"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98","Type":"ContainerStarted","Data":"2b83888e40e596804ac3f4bdcd3dc589634bec914b2bf8da5fe22c3591a4cb1f"} Jul 31 20:42:48.732738 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:48.732649 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564444965-4pjl8" event={"ID":"6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98","Type":"ContainerStarted","Data":"de5a738f1f1b497aea6979f93f79860d92160c7498a21d3afde4dfd115f3894f"} Jul 31 20:42:48.750118 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:42:48.750055 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7564444965-4pjl8" podStartSLOduration=1.03976159 podStartE2EDuration="1.750036661s" totalImagesPullingTime="710.275071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 20:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 20:42:48.748360719 +0000 UTC m=+747.124733899" watchObservedRunningTime="2026-07-31 20:42:48.750036661 +0000 UTC m=+747.126409840" Jul 31 20:43:17.979362 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:43:17.979316 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:43:27.586509 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:43:27.586471 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:43:40.680066 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:43:40.680021 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:45:19.282748 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:45:19.282707 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:45:29.684384 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:45:29.684343 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:46:27.785810 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:46:27.785765 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:46:34.882153 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:46:34.882117 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:46:50.482332 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:46:50.482250 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:46:57.480309 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:46:57.480270 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:06.486891 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:06.486828 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:12.987832 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:12.987792 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:28.186774 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:28.186734 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:33.586678 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:33.586632 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:42.389538 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:42.389499 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:47:51.290715 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:47:51.290669 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:48:03.590247 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:48:03.590204 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:48:14.377970 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:48:14.377887 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:48:36.882065 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:48:36.882024 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:48:47.387796 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:48:47.387754 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:48:52.685639 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:48:52.685600 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:00.289013 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:00.288979 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:09.095183 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:09.095147 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:13.483681 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:13.483643 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:19.782133 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:19.782099 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:26.078522 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:26.078483 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:39.985197 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:39.985159 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:47.680269 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:47.680181 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:49:58.190026 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:49:58.189991 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:50:03.290703 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:50:03.290665 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:50:09.488055 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:50:09.488015 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:50:18.278308 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:50:18.278253 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:50:23.388145 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:50:23.388101 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:50:34.178028 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:50:34.177992 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:52:50.181932 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:52:50.181827 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:52:57.089462 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:52:57.089424 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:53:18.379225 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:53:18.379178 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:53:28.982273 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:53:28.982235 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:53:35.586592 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:53:35.586552 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:53:44.582236 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:53:44.582200 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:53:50.482128 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:53:50.482082 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:00.180669 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:00.180631 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:06.483097 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:06.483051 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:15.475003 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:15.474916 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:21.086971 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:21.086931 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:31.484391 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:31.484355 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:42.372434 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:42.372400 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:54:49.381850 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:54:49.381808 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:55:11.087810 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:55:11.087764 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:55:16.082883 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:55:16.082841 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:55:26.193155 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:55:26.193117 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:55:43.996342 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:55:43.996261 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:02.081101 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:02.081057 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:13.882219 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:13.882180 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:22.083274 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:22.083233 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:34.083089 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:34.083054 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:41.386780 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:41.386735 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:51.182969 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:51.182931 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:56:58.176880 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:56:58.176833 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:10.886558 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:10.886521 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:16.077343 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:16.077266 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:31.184525 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:31.184485 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:38.681772 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:38.681723 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:52.286567 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:52.286530 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:57:58.576096 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:57:58.576056 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:06.971710 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:06.971675 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:12.985858 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:12.985821 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:19.395043 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:19.395006 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:28.891309 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:28.891270 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:35.989763 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:35.989702 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:45.980170 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:45.980085 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:58:53.181191 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:58:53.181149 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:03.185331 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:03.185292 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:18.175652 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:18.175614 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:24.287625 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:24.287572 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:30.785428 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:30.785391 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:42.280645 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:42.280592 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:50.485042 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:50.485006 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 20:59:56.180362 ip-10-0-141-177 kubenswrapper[2544]: I0731 20:59:56.180320 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:00:10.381584 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:00:10.381541 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:00:40.187654 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:00:40.187618 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:00:46.882277 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:00:46.882236 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:01:15.280945 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:01:15.280903 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:01:25.295884 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:01:25.295840 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:02:00.784761 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:02:00.784667 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:02:09.677572 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:02:09.677534 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:02:36.683306 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:02:36.683268 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:02:49.381109 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:02:49.381068 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:04:20.696297 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:04:20.696259 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:04:28.082070 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:04:28.082031 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:04:38.684955 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:04:38.684914 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:05:19.079944 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:05:19.079844 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:05:57.586056 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:05:57.586018 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:07:39.985544 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:07:39.985499 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:07:46.878650 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:07:46.878610 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:08:36.983637 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:08:36.983595 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:08:51.984341 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:08:51.984291 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:09:15.477732 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:09:15.477641 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:09:21.293194 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:09:21.293154 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:10:11.288352 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:10:11.288305 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:10:23.388590 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:10:23.388550 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:10:31.483510 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:10:31.483472 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:11:20.988070 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:11:20.988015 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:11:49.785037 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:11:49.785000 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:12:39.682967 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:12:39.682927 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:13:07.391489 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.391456 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:13:07.394892 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.394861 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.398718 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.398691 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c8d2ed\"/\"openshift-service-ca.crt\"" Jul 31 21:13:07.398718 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.398710 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c8d2ed\"/\"kube-root-ca.crt\"" Jul 31 21:13:07.398718 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.398694 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c8d2ed\"/\"test-model-c8d2ed-kserve-self-signed-certs\"" Jul 31 21:13:07.398965 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.398694 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c8d2ed\"/\"default-dockercfg-lgzkn\"" Jul 31 21:13:07.403147 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.403124 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:13:07.440773 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.440739 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.440929 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.440803 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.440929 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.440837 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkf59\" (UniqueName: \"kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.441009 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.440927 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.441009 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.440984 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.441072 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.441020 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.541734 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541693 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.541734 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541740 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542022 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541769 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542022 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541793 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542022 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541820 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkf59\" (UniqueName: \"kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542022 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.541974 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542273 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.542186 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542273 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.542224 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.542357 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.542280 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.544329 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.544299 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.544621 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.544604 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.549610 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.549586 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkf59\" (UniqueName: \"kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59\") pod \"test-model-c8d2ed-kserve-5b9b877c5c-pqg65\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.704834 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.704750 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:07.859461 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:07.859435 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:13:07.861844 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:13:07.861809 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod397c9dd5_e1a0_4055_8c82_242460810d96.slice/crio-284ca2b5b86bd0d1d0d18b98144c426570ada5093ec49bb78feb23e388bf468d WatchSource:0}: Error finding container 284ca2b5b86bd0d1d0d18b98144c426570ada5093ec49bb78feb23e388bf468d: Status 404 returned error can't find the container with id 284ca2b5b86bd0d1d0d18b98144c426570ada5093ec49bb78feb23e388bf468d Jul 31 21:13:08.766662 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:08.766626 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" event={"ID":"397c9dd5-e1a0-4055-8c82-242460810d96","Type":"ContainerStarted","Data":"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26"} Jul 31 21:13:08.766662 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:08.766670 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" event={"ID":"397c9dd5-e1a0-4055-8c82-242460810d96","Type":"ContainerStarted","Data":"284ca2b5b86bd0d1d0d18b98144c426570ada5093ec49bb78feb23e388bf468d"} Jul 31 21:13:08.782278 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:08.782227 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" podStartSLOduration=1.782205351 podStartE2EDuration="1.782205351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:13:08.780769274 +0000 UTC m=+2567.157142454" watchObservedRunningTime="2026-07-31 21:13:08.782205351 +0000 UTC m=+2567.158578532" Jul 31 21:13:11.785322 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:11.785285 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:13:17.704932 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:17.704887 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:17.705425 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:17.704956 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:17.718141 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:17.718112 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:17.827179 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:17.827147 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:13:52.015860 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:52.015823 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:13:54.380067 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:13:54.380028 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:14:01.385007 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:01.384962 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:14:07.036053 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.035988 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" podUID="397c9dd5-e1a0-4055-8c82-242460810d96" containerName="main" containerID="cri-o://8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26" gracePeriod=45 Jul 31 21:14:07.297556 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.297486 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:14:07.318561 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318533 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318723 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318645 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318723 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318679 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318723 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318708 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkf59\" (UniqueName: \"kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318917 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318743 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318917 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318797 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home\") pod \"397c9dd5-e1a0-4055-8c82-242460810d96\" (UID: \"397c9dd5-e1a0-4055-8c82-242460810d96\") " Jul 31 21:14:07.318917 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.318822 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache" (OuterVolumeSpecName: "model-cache") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:14:07.319202 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.319168 2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-model-cache\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:07.319351 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.319201 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:14:07.319420 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.319368 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home" (OuterVolumeSpecName: "home") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:14:07.321427 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.321404 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:14:07.321524 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.321434 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59" (OuterVolumeSpecName: "kube-api-access-lkf59") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "kube-api-access-lkf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:14:07.321642 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.321624 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm" (OuterVolumeSpecName: "dshm") pod "397c9dd5-e1a0-4055-8c82-242460810d96" (UID: "397c9dd5-e1a0-4055-8c82-242460810d96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:14:07.420077 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.420038 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkf59\" (UniqueName: \"kubernetes.io/projected/397c9dd5-e1a0-4055-8c82-242460810d96-kube-api-access-lkf59\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:07.420077 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.420069 2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397c9dd5-e1a0-4055-8c82-242460810d96-tls-certs\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:07.420077 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.420077 2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-home\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:07.420077 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.420085 2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-dshm\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:07.420347 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:07.420092 2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397c9dd5-e1a0-4055-8c82-242460810d96-tmp-dir\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:14:08.062361 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.062324 2544 generic.go:356] "Generic (PLEG): container finished" podID="397c9dd5-e1a0-4055-8c82-242460810d96" containerID="8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26" exitCode=0 Jul 31 21:14:08.062837 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.062388 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" Jul 31 21:14:08.062837 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.062417 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" event={"ID":"397c9dd5-e1a0-4055-8c82-242460810d96","Type":"ContainerDied","Data":"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26"} Jul 31 21:14:08.062837 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.062470 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65" event={"ID":"397c9dd5-e1a0-4055-8c82-242460810d96","Type":"ContainerDied","Data":"284ca2b5b86bd0d1d0d18b98144c426570ada5093ec49bb78feb23e388bf468d"} Jul 31 21:14:08.062837 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.062495 2544 scope.go:117] "RemoveContainer" containerID="8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26" Jul 31 21:14:08.073288 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.073267 2544 scope.go:117] "RemoveContainer" containerID="8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26" Jul 31 21:14:08.073653 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:14:08.073625 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26\": container with ID starting with 8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26 not found: ID does not exist" containerID="8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26" Jul 31 21:14:08.073724 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.073666 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26"} err="failed to get container status \"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26\": rpc error: code = NotFound desc = could not find container \"8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26\": container with ID starting with 8bbeb1149ec7b633dc573157be8d5bdf4de8a5ba48046bdfd76b41ff08035a26 not found: ID does not exist" Jul 31 21:14:08.084573 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.084541 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:14:08.088175 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.088150 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c8d2ed/test-model-c8d2ed-kserve-5b9b877c5c-pqg65"] Jul 31 21:14:08.113139 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:08.113112 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397c9dd5-e1a0-4055-8c82-242460810d96" path="/var/lib/kubelet/pods/397c9dd5-e1a0-4055-8c82-242460810d96/volumes" Jul 31 21:14:20.986498 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:20.986459 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:14:34.477056 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:14:34.477018 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:15:02.788143 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:15:02.788096 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:15:45.292236 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:15:45.292195 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:16:03.086364 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:16:03.086324 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:16:14.979266 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:16:14.979228 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:16:43.782711 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:16:43.782673 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:17:13.385585 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:13.385539 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:17:29.773452 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.773410 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:17:29.774033 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.774016 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397c9dd5-e1a0-4055-8c82-242460810d96" containerName="main" Jul 31 21:17:29.774085 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.774036 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="397c9dd5-e1a0-4055-8c82-242460810d96" containerName="main" Jul 31 21:17:29.774173 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.774151 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="397c9dd5-e1a0-4055-8c82-242460810d96" containerName="main" Jul 31 21:17:29.777183 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.777165 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.779950 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.779930 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6938c5\"/\"default-dockercfg-pxlj4\"" Jul 31 21:17:29.781008 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.780961 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6938c5\"/\"kube-root-ca.crt\"" Jul 31 21:17:29.781119 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.781009 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6938c5\"/\"openshift-service-ca.crt\"" Jul 31 21:17:29.781119 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.781047 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6938c5\"/\"auth-test-model-6938c5-kserve-self-signed-certs\"" Jul 31 21:17:29.785623 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.785597 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:17:29.844263 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844222 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.844263 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844263 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.844470 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844290 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.844470 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844384 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntkl\" (UniqueName: \"kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.844470 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844405 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.844470 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.844436 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945633 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945594 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945658 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945678 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945697 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945754 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntkl\" (UniqueName: \"kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.945819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.945773 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.946166 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.946144 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.946242 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.946197 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.946242 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.946231 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.948192 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.948165 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.948317 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.948300 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:29.956906 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:29.956851 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntkl\" (UniqueName: \"kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl\") pod \"auth-test-model-6938c5-kserve-5df96db7bc-chxs8\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:30.089026 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:30.088932 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:30.228779 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:30.228751 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:17:30.231544 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:17:30.231509 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20209d3d_681a_4990_bcc8_9d4ae1444188.slice/crio-60ff376f82c2aca2cc257ad4c07b40f501d15e506b6e48e45220382fd118cd02 WatchSource:0}: Error finding container 60ff376f82c2aca2cc257ad4c07b40f501d15e506b6e48e45220382fd118cd02: Status 404 returned error can't find the container with id 60ff376f82c2aca2cc257ad4c07b40f501d15e506b6e48e45220382fd118cd02 Jul 31 21:17:31.069768 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:31.069723 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" event={"ID":"20209d3d-681a-4990-bcc8-9d4ae1444188","Type":"ContainerStarted","Data":"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60"} Jul 31 21:17:31.069768 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:31.069777 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" event={"ID":"20209d3d-681a-4990-bcc8-9d4ae1444188","Type":"ContainerStarted","Data":"60ff376f82c2aca2cc257ad4c07b40f501d15e506b6e48e45220382fd118cd02"} Jul 31 21:17:31.087061 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:31.086987 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" podStartSLOduration=2.086971819 podStartE2EDuration="2.086971819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:17:31.085320067 +0000 UTC m=+2829.461693250" watchObservedRunningTime="2026-07-31 21:17:31.086971819 +0000 UTC m=+2829.463345071" Jul 31 21:17:32.580223 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:32.580183 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:17:40.089084 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:40.089044 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:40.089680 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:40.089106 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:40.101815 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:40.101787 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:17:40.124961 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:17:40.124928 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:18:19.886825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:18:19.886739 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:18:35.985613 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:18:35.985575 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:18:47.798764 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:18:47.798725 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:19:02.482784 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:19:02.482744 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:19:14.101599 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:19:14.101550 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:19:28.879988 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:19:28.879943 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:19:40.294891 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:19:40.294831 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:19:55.282881 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:19:55.282780 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:20:07.286616 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:07.286575 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:20:21.776635 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:21.776598 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:20:33.488014 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:33.487975 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:20:48.478080 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:48.478032 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:20:53.497927 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:53.497894 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:20:59.188068 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:20:59.188014 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:21:05.483240 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:05.483203 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:21:08.519710 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.519663 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" podUID="20209d3d-681a-4990-bcc8-9d4ae1444188" containerName="main" containerID="cri-o://24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60" gracePeriod=45 Jul 31 21:21:08.781086 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.781021 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:21:08.945391 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945351 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945428 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ntkl\" (UniqueName: \"kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945462 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945499 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945525 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945567 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir\") pod \"20209d3d-681a-4990-bcc8-9d4ae1444188\" (UID: \"20209d3d-681a-4990-bcc8-9d4ae1444188\") " Jul 31 21:21:08.945971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945653 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache" (OuterVolumeSpecName: "model-cache") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:21:08.945971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945747 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home" (OuterVolumeSpecName: "home") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:21:08.945971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945914 2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-model-cache\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:08.945971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945934 2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-home\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:08.945971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.945936 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:21:08.947963 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.947924 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm" (OuterVolumeSpecName: "dshm") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:21:08.948075 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.947971 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:21:08.948075 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:08.948005 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl" (OuterVolumeSpecName: "kube-api-access-9ntkl") pod "20209d3d-681a-4990-bcc8-9d4ae1444188" (UID: "20209d3d-681a-4990-bcc8-9d4ae1444188"). InnerVolumeSpecName "kube-api-access-9ntkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:21:09.047483 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.047387 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntkl\" (UniqueName: \"kubernetes.io/projected/20209d3d-681a-4990-bcc8-9d4ae1444188-kube-api-access-9ntkl\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:09.047483 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.047425 2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-dshm\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:09.047483 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.047437 2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20209d3d-681a-4990-bcc8-9d4ae1444188-tls-certs\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:09.047483 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.047447 2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20209d3d-681a-4990-bcc8-9d4ae1444188-tmp-dir\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:21:09.136292 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.136257 2544 generic.go:356] "Generic (PLEG): container finished" podID="20209d3d-681a-4990-bcc8-9d4ae1444188" containerID="24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60" exitCode=0 Jul 31 21:21:09.136484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.136306 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" Jul 31 21:21:09.136484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.136346 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" event={"ID":"20209d3d-681a-4990-bcc8-9d4ae1444188","Type":"ContainerDied","Data":"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60"} Jul 31 21:21:09.136484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.136399 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8" event={"ID":"20209d3d-681a-4990-bcc8-9d4ae1444188","Type":"ContainerDied","Data":"60ff376f82c2aca2cc257ad4c07b40f501d15e506b6e48e45220382fd118cd02"} Jul 31 21:21:09.136484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.136423 2544 scope.go:117] "RemoveContainer" containerID="24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60" Jul 31 21:21:09.146159 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.146135 2544 scope.go:117] "RemoveContainer" containerID="24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60" Jul 31 21:21:09.146468 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:21:09.146447 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60\": container with ID starting with 24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60 not found: ID does not exist" containerID="24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60" Jul 31 21:21:09.146544 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.146476 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60"} err="failed to get container status \"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60\": rpc error: code = NotFound desc = could not find container \"24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60\": container with ID starting with 24df3dad8e8c2d8786a11d4ee19f6590a54bd2221fe4c2c6b6ca7b6857652c60 not found: ID does not exist" Jul 31 21:21:09.157969 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.157933 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:21:09.161575 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:09.161546 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6938c5/auth-test-model-6938c5-kserve-5df96db7bc-chxs8"] Jul 31 21:21:10.113963 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:10.113929 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20209d3d-681a-4990-bcc8-9d4ae1444188" path="/var/lib/kubelet/pods/20209d3d-681a-4990-bcc8-9d4ae1444188/volumes" Jul 31 21:21:19.790699 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:21:19.790611 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:22:00.678088 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:00.678054 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:22:11.581108 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:11.581069 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:22:39.794767 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:39.794727 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:22:50.280757 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:50.280668 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:22:56.892536 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.892491 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:22:56.893135 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.893113 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20209d3d-681a-4990-bcc8-9d4ae1444188" containerName="main" Jul 31 21:22:56.893215 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.893139 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="20209d3d-681a-4990-bcc8-9d4ae1444188" containerName="main" Jul 31 21:22:56.893273 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.893228 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="20209d3d-681a-4990-bcc8-9d4ae1444188" containerName="main" Jul 31 21:22:56.896162 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.896139 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.898850 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.898828 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8de79c\"/\"kube-root-ca.crt\"" Jul 31 21:22:56.898850 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.898830 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8de79c\"/\"openshift-service-ca.crt\"" Jul 31 21:22:56.899857 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.899833 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8de79c\"/\"default-dockercfg-79vh2\"" Jul 31 21:22:56.899857 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.899847 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8de79c\"/\"rate-test-model-8de79c-kserve-self-signed-certs\"" Jul 31 21:22:56.904546 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.904520 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:22:56.967624 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967590 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.967793 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967637 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.967793 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967722 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.967793 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967758 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hml8k\" (UniqueName: \"kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.967915 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967824 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:56.967915 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:56.967840 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.068815 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.068770 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069024 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.068831 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hml8k\" (UniqueName: \"kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069024 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.068966 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069024 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.068992 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069129 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.069060 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069129 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.069095 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069459 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.069430 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069563 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.069443 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.069563 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.069481 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.071372 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.071340 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.071547 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.071536 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.077505 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.077473 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hml8k\" (UniqueName: \"kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k\") pod \"rate-test-model-8de79c-kserve-7574d486c9-qj2hq\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.207350 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.207242 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:22:57.349404 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.349377 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:22:57.351692 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:22:57.351655 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7704b401_0b13_4456_8a84_25303150a49b.slice/crio-8fe9149015aabc328b04a69708525bce34bbb3540c3abe91bc14b293a0ca49fd WatchSource:0}: Error finding container 8fe9149015aabc328b04a69708525bce34bbb3540c3abe91bc14b293a0ca49fd: Status 404 returned error can't find the container with id 8fe9149015aabc328b04a69708525bce34bbb3540c3abe91bc14b293a0ca49fd Jul 31 21:22:57.693613 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.693566 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" event={"ID":"7704b401-0b13-4456-8a84-25303150a49b","Type":"ContainerStarted","Data":"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d"} Jul 31 21:22:57.693835 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.693621 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" event={"ID":"7704b401-0b13-4456-8a84-25303150a49b","Type":"ContainerStarted","Data":"8fe9149015aabc328b04a69708525bce34bbb3540c3abe91bc14b293a0ca49fd"} Jul 31 21:22:57.711285 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:57.711228 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" podStartSLOduration=1.711211147 podStartE2EDuration="1.711211147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:22:57.709829606 +0000 UTC m=+3156.086202787" watchObservedRunningTime="2026-07-31 21:22:57.711211147 +0000 UTC m=+3156.087584325" Jul 31 21:22:59.579982 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:22:59.579942 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:23:07.207895 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:23:07.207827 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:23:07.207895 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:23:07.207906 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:23:07.220756 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:23:07.220722 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:23:07.754157 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:23:07.754126 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:23:28.081314 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:23:28.081269 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:24:02.679098 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:02.679002 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:24:11.291885 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:11.291821 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:24:26.888328 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:26.888288 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:24:32.405024 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:32.404986 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:24:36.280053 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:36.280014 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:24:40.325586 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.325552 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:40.329112 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.329094 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:40.331459 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.331434 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-plrtc\"" Jul 31 21:24:40.335719 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.335689 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:40.391938 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.391907 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7r4m\" (UniqueName: \"kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m\") pod \"test-curl-7674-139768272473232\" (UID: \"56c3b0b1-3a19-4d74-98c2-6b885da66c68\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:40.492469 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.492427 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7r4m\" (UniqueName: \"kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m\") pod \"test-curl-7674-139768272473232\" (UID: \"56c3b0b1-3a19-4d74-98c2-6b885da66c68\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:40.501272 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.501241 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7r4m\" (UniqueName: \"kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m\") pod \"test-curl-7674-139768272473232\" (UID: \"56c3b0b1-3a19-4d74-98c2-6b885da66c68\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:40.640664 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.640629 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:40.776291 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.776265 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:40.778927 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:24:40.778897 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56c3b0b1_3a19_4d74_98c2_6b885da66c68.slice/crio-50e490b28a59132da8ecb0de5ad24f075703f174dd2511e7bf103ae5d8c602e3 WatchSource:0}: Error finding container 50e490b28a59132da8ecb0de5ad24f075703f174dd2511e7bf103ae5d8c602e3: Status 404 returned error can't find the container with id 50e490b28a59132da8ecb0de5ad24f075703f174dd2511e7bf103ae5d8c602e3 Jul 31 21:24:40.780248 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:40.780229 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 21:24:41.222140 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:41.222098 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272473232" event={"ID":"56c3b0b1-3a19-4d74-98c2-6b885da66c68","Type":"ContainerStarted","Data":"50e490b28a59132da8ecb0de5ad24f075703f174dd2511e7bf103ae5d8c602e3"} Jul 31 21:24:42.228117 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:42.228082 2544 generic.go:356] "Generic (PLEG): container finished" podID="56c3b0b1-3a19-4d74-98c2-6b885da66c68" containerID="b8f4f223686147468f49deaea3bbb5bd1d6e85c81ad384094d963abece001ffd" exitCode=0 Jul 31 21:24:42.228511 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:42.228128 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272473232" event={"ID":"56c3b0b1-3a19-4d74-98c2-6b885da66c68","Type":"ContainerDied","Data":"b8f4f223686147468f49deaea3bbb5bd1d6e85c81ad384094d963abece001ffd"} Jul 31 21:24:43.376001 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.375974 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:43.408226 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.408195 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7674-139768272473232_56c3b0b1-3a19-4d74-98c2-6b885da66c68/test-curl-7674-139768272473232/0.log" Jul 31 21:24:43.421259 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.421231 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7r4m\" (UniqueName: \"kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m\") pod \"56c3b0b1-3a19-4d74-98c2-6b885da66c68\" (UID: \"56c3b0b1-3a19-4d74-98c2-6b885da66c68\") " Jul 31 21:24:43.423609 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.423585 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m" (OuterVolumeSpecName: "kube-api-access-t7r4m") pod "56c3b0b1-3a19-4d74-98c2-6b885da66c68" (UID: "56c3b0b1-3a19-4d74-98c2-6b885da66c68"). InnerVolumeSpecName "kube-api-access-t7r4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:24:43.431204 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.431178 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:43.434770 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.434744 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:43.522078 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.521985 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7r4m\" (UniqueName: \"kubernetes.io/projected/56c3b0b1-3a19-4d74-98c2-6b885da66c68-kube-api-access-t7r4m\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:43.555130 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.555094 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:43.555548 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.555533 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c3b0b1-3a19-4d74-98c2-6b885da66c68" containerName="test-curl-7674-139768272473232" Jul 31 21:24:43.555592 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.555550 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c3b0b1-3a19-4d74-98c2-6b885da66c68" containerName="test-curl-7674-139768272473232" Jul 31 21:24:43.555641 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.555632 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c3b0b1-3a19-4d74-98c2-6b885da66c68" containerName="test-curl-7674-139768272473232" Jul 31 21:24:43.558809 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.558793 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:43.564845 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.564812 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:43.623494 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.623447 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpqw\" (UniqueName: \"kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw\") pod \"test-curl-7674-139768272473232\" (UID: \"10ce12ad-4079-4239-a9f9-2eb962efe8e1\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:43.724322 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.724282 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpqw\" (UniqueName: \"kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw\") pod \"test-curl-7674-139768272473232\" (UID: \"10ce12ad-4079-4239-a9f9-2eb962efe8e1\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:43.732785 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.732747 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpqw\" (UniqueName: \"kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw\") pod \"test-curl-7674-139768272473232\" (UID: \"10ce12ad-4079-4239-a9f9-2eb962efe8e1\") " pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:43.869236 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:43.869197 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:44.007629 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:44.007595 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:44.010727 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:24:44.010682 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10ce12ad_4079_4239_a9f9_2eb962efe8e1.slice/crio-4126123a3a2c3ea50ab089b88caf6c826a882b5659e9919e296e7bb18a8fa072 WatchSource:0}: Error finding container 4126123a3a2c3ea50ab089b88caf6c826a882b5659e9919e296e7bb18a8fa072: Status 404 returned error can't find the container with id 4126123a3a2c3ea50ab089b88caf6c826a882b5659e9919e296e7bb18a8fa072 Jul 31 21:24:44.113406 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:44.113373 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c3b0b1-3a19-4d74-98c2-6b885da66c68" path="/var/lib/kubelet/pods/56c3b0b1-3a19-4d74-98c2-6b885da66c68/volumes" Jul 31 21:24:44.239134 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:44.239100 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:44.239327 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:44.239100 2544 scope.go:117] "RemoveContainer" containerID="b8f4f223686147468f49deaea3bbb5bd1d6e85c81ad384094d963abece001ffd" Jul 31 21:24:44.240494 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:44.240467 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272473232" event={"ID":"10ce12ad-4079-4239-a9f9-2eb962efe8e1","Type":"ContainerStarted","Data":"4126123a3a2c3ea50ab089b88caf6c826a882b5659e9919e296e7bb18a8fa072"} Jul 31 21:24:45.247993 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:45.247948 2544 generic.go:356] "Generic (PLEG): container finished" podID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" containerID="f8d0b975806e131bf4c513cc480ecaaef8350dfb34d8b973812948cb91cd566d" exitCode=0 Jul 31 21:24:45.248416 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:45.248027 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272473232" event={"ID":"10ce12ad-4079-4239-a9f9-2eb962efe8e1","Type":"ContainerDied","Data":"f8d0b975806e131bf4c513cc480ecaaef8350dfb34d8b973812948cb91cd566d"} Jul 31 21:24:46.395575 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.395550 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:46.414010 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.413981 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7674-139768272473232_10ce12ad-4079-4239-a9f9-2eb962efe8e1/test-curl-7674-139768272473232/0.log" Jul 31 21:24:46.434600 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.434567 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:46.437727 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.437704 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-139768272473232"] Jul 31 21:24:46.450631 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.450604 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbpqw\" (UniqueName: \"kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw\") pod \"10ce12ad-4079-4239-a9f9-2eb962efe8e1\" (UID: \"10ce12ad-4079-4239-a9f9-2eb962efe8e1\") " Jul 31 21:24:46.452817 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.452788 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw" (OuterVolumeSpecName: "kube-api-access-fbpqw") pod "10ce12ad-4079-4239-a9f9-2eb962efe8e1" (UID: "10ce12ad-4079-4239-a9f9-2eb962efe8e1"). InnerVolumeSpecName "kube-api-access-fbpqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:24:46.552334 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.552247 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbpqw\" (UniqueName: \"kubernetes.io/projected/10ce12ad-4079-4239-a9f9-2eb962efe8e1-kube-api-access-fbpqw\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:46.552819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.552798 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7674-139768272482352"] Jul 31 21:24:46.553226 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.553212 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" containerName="test-curl-7674-139768272473232" Jul 31 21:24:46.553262 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.553229 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" containerName="test-curl-7674-139768272473232" Jul 31 21:24:46.553310 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.553301 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" containerName="test-curl-7674-139768272473232" Jul 31 21:24:46.556301 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.556286 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:46.562024 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.561996 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272482352"] Jul 31 21:24:46.653739 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.653697 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpl5\" (UniqueName: \"kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5\") pod \"test-curl-7674-139768272482352\" (UID: \"521c9faf-57a7-49eb-97a1-9fcdcf5f883a\") " pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:46.754475 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.754438 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpl5\" (UniqueName: \"kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5\") pod \"test-curl-7674-139768272482352\" (UID: \"521c9faf-57a7-49eb-97a1-9fcdcf5f883a\") " pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:46.762636 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.762611 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpl5\" (UniqueName: \"kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5\") pod \"test-curl-7674-139768272482352\" (UID: \"521c9faf-57a7-49eb-97a1-9fcdcf5f883a\") " pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:46.867014 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:46.866975 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:47.002448 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.002413 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7674-139768272482352"] Jul 31 21:24:47.004884 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:24:47.004849 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521c9faf_57a7_49eb_97a1_9fcdcf5f883a.slice/crio-4448c3b0f9e915893cadc5732ece946bcedf534e19e7080934a0a9a3939b0fc2 WatchSource:0}: Error finding container 4448c3b0f9e915893cadc5732ece946bcedf534e19e7080934a0a9a3939b0fc2: Status 404 returned error can't find the container with id 4448c3b0f9e915893cadc5732ece946bcedf534e19e7080934a0a9a3939b0fc2 Jul 31 21:24:47.260689 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.260602 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272482352" event={"ID":"521c9faf-57a7-49eb-97a1-9fcdcf5f883a","Type":"ContainerStarted","Data":"4448c3b0f9e915893cadc5732ece946bcedf534e19e7080934a0a9a3939b0fc2"} Jul 31 21:24:47.261807 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.261782 2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4126123a3a2c3ea50ab089b88caf6c826a882b5659e9919e296e7bb18a8fa072" Jul 31 21:24:47.261807 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.261808 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:47.272071 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:24:47.272044 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-139768272473232\" is forbidden: User \"system:node:ip-10-0-141-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-177.ec2.internal' and this object" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:47.425825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.425777 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" podUID="7704b401-0b13-4456-8a84-25303150a49b" containerName="main" containerID="cri-o://50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d" gracePeriod=45 Jul 31 21:24:47.699648 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.699619 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:24:47.701981 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:24:47.701951 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7674-139768272473232\" is forbidden: User \"system:node:ip-10-0-141-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-177.ec2.internal' and this object" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" pod="opendatahub/test-curl-7674-139768272473232" Jul 31 21:24:47.765168 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765141 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765342 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765179 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765342 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765222 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765342 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765250 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765342 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765315 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hml8k\" (UniqueName: \"kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765570 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765346 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home\") pod \"7704b401-0b13-4456-8a84-25303150a49b\" (UID: \"7704b401-0b13-4456-8a84-25303150a49b\") " Jul 31 21:24:47.765570 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765450 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:24:47.765570 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765461 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:24:47.765702 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765655 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home" (OuterVolumeSpecName: "home") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:24:47.765702 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765658 2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-tmp-dir\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:47.765819 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.765702 2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-model-cache\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:47.767720 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.767691 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:24:47.767720 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.767710 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k" (OuterVolumeSpecName: "kube-api-access-hml8k") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "kube-api-access-hml8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:24:47.767906 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.767736 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm" (OuterVolumeSpecName: "dshm") pod "7704b401-0b13-4456-8a84-25303150a49b" (UID: "7704b401-0b13-4456-8a84-25303150a49b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:24:47.866265 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.866230 2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-dshm\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:47.866424 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.866282 2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7704b401-0b13-4456-8a84-25303150a49b-tls-certs\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:47.866424 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.866295 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hml8k\" (UniqueName: \"kubernetes.io/projected/7704b401-0b13-4456-8a84-25303150a49b-kube-api-access-hml8k\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:47.866424 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:47.866303 2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7704b401-0b13-4456-8a84-25303150a49b-home\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:48.114658 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.114619 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ce12ad-4079-4239-a9f9-2eb962efe8e1" path="/var/lib/kubelet/pods/10ce12ad-4079-4239-a9f9-2eb962efe8e1/volumes" Jul 31 21:24:48.268028 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.267995 2544 generic.go:356] "Generic (PLEG): container finished" podID="521c9faf-57a7-49eb-97a1-9fcdcf5f883a" containerID="79497c9675172640ee62d269193fc25344fdc2cea702e8decde5a8cc325a9760" exitCode=0 Jul 31 21:24:48.268187 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.268098 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7674-139768272482352" event={"ID":"521c9faf-57a7-49eb-97a1-9fcdcf5f883a","Type":"ContainerDied","Data":"79497c9675172640ee62d269193fc25344fdc2cea702e8decde5a8cc325a9760"} Jul 31 21:24:48.269406 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.269380 2544 generic.go:356] "Generic (PLEG): container finished" podID="7704b401-0b13-4456-8a84-25303150a49b" containerID="50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d" exitCode=0 Jul 31 21:24:48.269552 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.269428 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" event={"ID":"7704b401-0b13-4456-8a84-25303150a49b","Type":"ContainerDied","Data":"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d"} Jul 31 21:24:48.269552 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.269438 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" Jul 31 21:24:48.269552 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.269455 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq" event={"ID":"7704b401-0b13-4456-8a84-25303150a49b","Type":"ContainerDied","Data":"8fe9149015aabc328b04a69708525bce34bbb3540c3abe91bc14b293a0ca49fd"} Jul 31 21:24:48.269552 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.269480 2544 scope.go:117] "RemoveContainer" containerID="50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d" Jul 31 21:24:48.278356 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.278228 2544 scope.go:117] "RemoveContainer" containerID="50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d" Jul 31 21:24:48.278528 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:24:48.278507 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d\": container with ID starting with 50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d not found: ID does not exist" containerID="50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d" Jul 31 21:24:48.278575 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.278551 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d"} err="failed to get container status \"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d\": rpc error: code = NotFound desc = could not find container \"50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d\": container with ID starting with 50cd387ab06a83a4ef741796fc18dec87f17f71125e902ec2878afdefcc6c54d not found: ID does not exist" Jul 31 21:24:48.293803 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.293731 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:24:48.295483 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:48.295464 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8de79c/rate-test-model-8de79c-kserve-7574d486c9-qj2hq"] Jul 31 21:24:49.415545 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.415518 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:49.433332 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.433304 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7674-139768272482352_521c9faf-57a7-49eb-97a1-9fcdcf5f883a/test-curl-7674-139768272482352/0.log" Jul 31 21:24:49.454993 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.454958 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7674-139768272482352"] Jul 31 21:24:49.460067 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.460038 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7674-139768272482352"] Jul 31 21:24:49.481297 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.481269 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnpl5\" (UniqueName: \"kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5\") pod \"521c9faf-57a7-49eb-97a1-9fcdcf5f883a\" (UID: \"521c9faf-57a7-49eb-97a1-9fcdcf5f883a\") " Jul 31 21:24:49.483497 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.483470 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5" (OuterVolumeSpecName: "kube-api-access-wnpl5") pod "521c9faf-57a7-49eb-97a1-9fcdcf5f883a" (UID: "521c9faf-57a7-49eb-97a1-9fcdcf5f883a"). InnerVolumeSpecName "kube-api-access-wnpl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:24:49.582887 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:49.582795 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpl5\" (UniqueName: \"kubernetes.io/projected/521c9faf-57a7-49eb-97a1-9fcdcf5f883a-kube-api-access-wnpl5\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:24:50.115344 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:50.115306 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="521c9faf-57a7-49eb-97a1-9fcdcf5f883a" path="/var/lib/kubelet/pods/521c9faf-57a7-49eb-97a1-9fcdcf5f883a/volumes" Jul 31 21:24:50.115882 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:50.115848 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7704b401-0b13-4456-8a84-25303150a49b" path="/var/lib/kubelet/pods/7704b401-0b13-4456-8a84-25303150a49b/volumes" Jul 31 21:24:50.148556 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:24:50.148520 2544 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521c9faf_57a7_49eb_97a1_9fcdcf5f883a.slice\": RecentStats: unable to find data in memory cache]" Jul 31 21:24:50.148711 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:24:50.148536 2544 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521c9faf_57a7_49eb_97a1_9fcdcf5f883a.slice\": RecentStats: unable to find data in memory cache]" Jul 31 21:24:50.281334 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:50.281305 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7674-139768272482352" Jul 31 21:24:50.281492 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:24:50.281350 2544 scope.go:117] "RemoveContainer" containerID="79497c9675172640ee62d269193fc25344fdc2cea702e8decde5a8cc325a9760" Jul 31 21:25:00.490993 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:00.490956 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:25:26.419365 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419328 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:25:26.419803 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419788 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="521c9faf-57a7-49eb-97a1-9fcdcf5f883a" containerName="test-curl-7674-139768272482352" Jul 31 21:25:26.419803 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419803 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="521c9faf-57a7-49eb-97a1-9fcdcf5f883a" containerName="test-curl-7674-139768272482352" Jul 31 21:25:26.419901 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419821 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7704b401-0b13-4456-8a84-25303150a49b" containerName="main" Jul 31 21:25:26.419901 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419827 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7704b401-0b13-4456-8a84-25303150a49b" containerName="main" Jul 31 21:25:26.419971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419919 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7704b401-0b13-4456-8a84-25303150a49b" containerName="main" Jul 31 21:25:26.419971 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.419935 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="521c9faf-57a7-49eb-97a1-9fcdcf5f883a" containerName="test-curl-7674-139768272482352" Jul 31 21:25:26.424861 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.424840 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.427581 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.427556 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 21:25:26.427731 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.427608 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 31 21:25:26.427731 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.427565 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c3fd2-serving-cert\"" Jul 31 21:25:26.428569 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.428549 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-j44wr\"" Jul 31 21:25:26.433192 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.432899 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:25:26.552070 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.552026 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.552070 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.552075 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlwn\" (UniqueName: \"kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.552288 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.552157 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.652723 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.652685 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.653000 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.652738 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlwn\" (UniqueName: \"kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.653000 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.652802 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.654006 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.653974 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.661512 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.661483 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.664055 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.664021 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlwn\" (UniqueName: \"kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn\") pod \"maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.738098 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.737998 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:26.877160 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:26.877133 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:25:26.879918 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:25:26.879887 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910463bc_3500_439c_81d3_3423cdb2cdc9.slice/crio-074c06d3b75c5766cd4dc4bb76a748b714889cd21903434a059da47fea01c498 WatchSource:0}: Error finding container 074c06d3b75c5766cd4dc4bb76a748b714889cd21903434a059da47fea01c498: Status 404 returned error can't find the container with id 074c06d3b75c5766cd4dc4bb76a748b714889cd21903434a059da47fea01c498 Jul 31 21:25:27.473293 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:27.473220 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" event={"ID":"910463bc-3500-439c-81d3-3423cdb2cdc9","Type":"ContainerStarted","Data":"074c06d3b75c5766cd4dc4bb76a748b714889cd21903434a059da47fea01c498"} Jul 31 21:25:29.485734 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:29.485689 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" event={"ID":"910463bc-3500-439c-81d3-3423cdb2cdc9","Type":"ContainerStarted","Data":"0ccd2e2ae809e1a052b41f093ccc0376876e5807c36b4d5cb8a6983d00e94cab"} Jul 31 21:25:29.486206 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:29.485831 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:25:29.502449 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:29.502396 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" podStartSLOduration=1.363106809 podStartE2EDuration="3.502380104s" totalImagesPullingTime="2.139273295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:25:29.501668863 +0000 UTC m=+3307.878042046" watchObservedRunningTime="2026-07-31 21:25:29.502380104 +0000 UTC m=+3307.878753283" Jul 31 21:25:29.683743 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:29.683704 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:25:35.496860 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:25:35.496827 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:26:04.882183 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.882145 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:26:04.885972 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.885948 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:04.888536 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.888507 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4cb74\"/\"openshift-service-ca.crt\"" Jul 31 21:26:04.889450 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.889430 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4cb74\"/\"default-dockercfg-vgrh4\"" Jul 31 21:26:04.889552 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.889487 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4cb74\"/\"kube-root-ca.crt\"" Jul 31 21:26:04.889622 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.889548 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4cb74\"/\"ipp-route-b4cb74-kserve-self-signed-certs\"" Jul 31 21:26:04.893541 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:04.893514 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:26:05.024264 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024221 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.024473 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024278 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.024473 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024366 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtksd\" (UniqueName: \"kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.024473 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024390 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.024473 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024419 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.024629 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.024499 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125619 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125579 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125649 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125683 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125733 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtksd\" (UniqueName: \"kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125751 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.125825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.125773 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.126142 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.126077 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.126211 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.126168 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.126211 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.126194 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.128221 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.128191 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.128469 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.128450 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.133606 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.133540 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtksd\" (UniqueName: \"kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd\") pod \"ipp-route-b4cb74-kserve-777cb7696c-xr8br\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.196620 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.196563 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:05.337590 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.337566 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:26:05.339995 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:26:05.339967 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7550496_b283_48c0_ad55_2d426a0c9e7c.slice/crio-2d561075646a15d634700f478acc029440c2a51ce2459aedc0ebd4556e014ac7 WatchSource:0}: Error finding container 2d561075646a15d634700f478acc029440c2a51ce2459aedc0ebd4556e014ac7: Status 404 returned error can't find the container with id 2d561075646a15d634700f478acc029440c2a51ce2459aedc0ebd4556e014ac7 Jul 31 21:26:05.671440 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.671338 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" event={"ID":"c7550496-b283-48c0-ad55-2d426a0c9e7c","Type":"ContainerStarted","Data":"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a"} Jul 31 21:26:05.671440 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.671386 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" event={"ID":"c7550496-b283-48c0-ad55-2d426a0c9e7c","Type":"ContainerStarted","Data":"2d561075646a15d634700f478acc029440c2a51ce2459aedc0ebd4556e014ac7"} Jul 31 21:26:05.692942 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:05.692891 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" podStartSLOduration=1.6928579849999998 podStartE2EDuration="1.692857985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:26:05.690252383 +0000 UTC m=+3344.066625567" watchObservedRunningTime="2026-07-31 21:26:05.692857985 +0000 UTC m=+3344.069231162" Jul 31 21:26:06.779229 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:06.779188 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:26:15.197574 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:15.197535 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:15.197999 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:15.197594 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:15.210204 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:15.210174 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:15.732939 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:15.732909 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:26:30.890560 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:30.890515 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:26:46.660967 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.660928 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:26:46.664424 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.664406 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.666998 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.666976 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-69e90d-serving-cert\"" Jul 31 21:26:46.674516 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.674486 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:26:46.737590 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.737557 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lcj\" (UniqueName: \"kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.737775 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.737643 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.737775 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.737676 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.838463 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.838426 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.838638 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.838473 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.838638 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.838532 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lcj\" (UniqueName: \"kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.838717 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:26:46.838654 2544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-serving-cert: secret "maas-api-e2e-ipp-cleanup-69e90d-serving-cert" not found Jul 31 21:26:46.838752 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:26:46.838724 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls podName:db85565c-4f7f-4265-a86f-371e0cf8a33d nodeName:}" failed. No retries permitted until 2026-07-31 21:26:47.338703215 +0000 UTC m=+3385.715076378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" (UID: "db85565c-4f7f-4265-a86f-371e0cf8a33d") : secret "maas-api-e2e-ipp-cleanup-69e90d-serving-cert" not found Jul 31 21:26:46.839158 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.839138 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:46.849817 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:46.849790 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lcj\" (UniqueName: \"kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:47.343341 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:47.343300 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:47.346147 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:47.346113 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:47.577109 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:47.577062 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:47.720127 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:47.720102 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:26:47.722565 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:26:47.722529 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb85565c_4f7f_4265_a86f_371e0cf8a33d.slice/crio-aa2d413d372bcc8efc745e3d55024c54a81f4a3c353aee4d059b8ab879afe397 WatchSource:0}: Error finding container aa2d413d372bcc8efc745e3d55024c54a81f4a3c353aee4d059b8ab879afe397: Status 404 returned error can't find the container with id aa2d413d372bcc8efc745e3d55024c54a81f4a3c353aee4d059b8ab879afe397 Jul 31 21:26:47.889487 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:47.889388 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" event={"ID":"db85565c-4f7f-4265-a86f-371e0cf8a33d","Type":"ContainerStarted","Data":"aa2d413d372bcc8efc745e3d55024c54a81f4a3c353aee4d059b8ab879afe397"} Jul 31 21:26:48.896043 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:48.896002 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" event={"ID":"db85565c-4f7f-4265-a86f-371e0cf8a33d","Type":"ContainerStarted","Data":"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be"} Jul 31 21:26:48.896499 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:48.896119 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:26:48.914830 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:48.914779 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" podStartSLOduration=2.579118538 podStartE2EDuration="2.914762686s" totalImagesPullingTime="335.644148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 21:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:26:48.913313292 +0000 UTC m=+3387.289686476" watchObservedRunningTime="2026-07-31 21:26:48.914762686 +0000 UTC m=+3387.291135918" Jul 31 21:26:51.780004 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:51.779960 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:26:54.907048 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:26:54.907005 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:27:17.118190 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.118092 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:27:17.118683 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.118330 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" podUID="db85565c-4f7f-4265-a86f-371e0cf8a33d" containerName="maas-api" containerID="cri-o://560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be" gracePeriod=30 Jul 31 21:27:17.379670 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.379607 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:27:17.432430 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.432389 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") pod \"db85565c-4f7f-4265-a86f-371e0cf8a33d\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " Jul 31 21:27:17.432617 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.432516 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96lcj\" (UniqueName: \"kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj\") pod \"db85565c-4f7f-4265-a86f-371e0cf8a33d\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " Jul 31 21:27:17.432688 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.432616 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca\") pod \"db85565c-4f7f-4265-a86f-371e0cf8a33d\" (UID: \"db85565c-4f7f-4265-a86f-371e0cf8a33d\") " Jul 31 21:27:17.432977 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.432948 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db85565c-4f7f-4265-a86f-371e0cf8a33d" (UID: "db85565c-4f7f-4265-a86f-371e0cf8a33d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 21:27:17.434852 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.434821 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj" (OuterVolumeSpecName: "kube-api-access-96lcj") pod "db85565c-4f7f-4265-a86f-371e0cf8a33d" (UID: "db85565c-4f7f-4265-a86f-371e0cf8a33d"). InnerVolumeSpecName "kube-api-access-96lcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:27:17.434852 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.434834 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db85565c-4f7f-4265-a86f-371e0cf8a33d" (UID: "db85565c-4f7f-4265-a86f-371e0cf8a33d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:27:17.533881 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.533829 2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db85565c-4f7f-4265-a86f-371e0cf8a33d-maas-api-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:27:17.533881 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.533878 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lcj\" (UniqueName: \"kubernetes.io/projected/db85565c-4f7f-4265-a86f-371e0cf8a33d-kube-api-access-96lcj\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:27:17.534082 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:17.533892 2544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db85565c-4f7f-4265-a86f-371e0cf8a33d-openshift-service-ca\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:27:18.043725 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.043685 2544 generic.go:356] "Generic (PLEG): container finished" podID="db85565c-4f7f-4265-a86f-371e0cf8a33d" containerID="560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be" exitCode=0 Jul 31 21:27:18.043930 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.043739 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" Jul 31 21:27:18.043930 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.043777 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" event={"ID":"db85565c-4f7f-4265-a86f-371e0cf8a33d","Type":"ContainerDied","Data":"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be"} Jul 31 21:27:18.043930 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.043852 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm" event={"ID":"db85565c-4f7f-4265-a86f-371e0cf8a33d","Type":"ContainerDied","Data":"aa2d413d372bcc8efc745e3d55024c54a81f4a3c353aee4d059b8ab879afe397"} Jul 31 21:27:18.043930 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.043898 2544 scope.go:117] "RemoveContainer" containerID="560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be" Jul 31 21:27:18.053625 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.053602 2544 scope.go:117] "RemoveContainer" containerID="560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be" Jul 31 21:27:18.053957 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:27:18.053937 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be\": container with ID starting with 560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be not found: ID does not exist" containerID="560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be" Jul 31 21:27:18.054019 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.053967 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be"} err="failed to get container status \"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be\": rpc error: code = NotFound desc = could not find container \"560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be\": container with ID starting with 560b8f0cec32bcb41f73e0a33fe0562d20725afa1f8c9d3e8f000c7869ae66be not found: ID does not exist" Jul 31 21:27:18.064319 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.064281 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:27:18.067695 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.067668 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-69e90d-66898786c6-594jm"] Jul 31 21:27:18.114080 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:18.114044 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db85565c-4f7f-4265-a86f-371e0cf8a33d" path="/var/lib/kubelet/pods/db85565c-4f7f-4265-a86f-371e0cf8a33d/volumes" Jul 31 21:27:20.186350 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:20.186313 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:27:40.782596 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:40.782557 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:27:48.476791 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:48.476752 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:27:50.084135 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:27:50.084099 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:28:03.497733 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.497681 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" podUID="c7550496-b283-48c0-ad55-2d426a0c9e7c" containerName="main" containerID="cri-o://56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a" gracePeriod=45 Jul 31 21:28:03.751278 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.751218 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:28:03.908059 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908021 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtksd\" (UniqueName: \"kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908312 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908108 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908312 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908177 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908312 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908270 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908312 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908302 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908604 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908323 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home\") pod \"c7550496-b283-48c0-ad55-2d426a0c9e7c\" (UID: \"c7550496-b283-48c0-ad55-2d426a0c9e7c\") " Jul 31 21:28:03.908604 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908406 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:28:03.908604 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908537 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:28:03.908604 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908590 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home" (OuterVolumeSpecName: "home") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:28:03.908851 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908748 2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-model-cache\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:03.908851 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908761 2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-tmp-dir\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:03.908851 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.908769 2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-home\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:03.910607 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.910579 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm" (OuterVolumeSpecName: "dshm") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 21:28:03.910607 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.910589 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:28:03.910738 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:03.910648 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd" (OuterVolumeSpecName: "kube-api-access-jtksd") pod "c7550496-b283-48c0-ad55-2d426a0c9e7c" (UID: "c7550496-b283-48c0-ad55-2d426a0c9e7c"). InnerVolumeSpecName "kube-api-access-jtksd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:28:04.009931 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.009808 2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7550496-b283-48c0-ad55-2d426a0c9e7c-dshm\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:04.009931 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.009841 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtksd\" (UniqueName: \"kubernetes.io/projected/c7550496-b283-48c0-ad55-2d426a0c9e7c-kube-api-access-jtksd\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:04.009931 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.009853 2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7550496-b283-48c0-ad55-2d426a0c9e7c-tls-certs\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:28:04.293728 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.293643 2544 generic.go:356] "Generic (PLEG): container finished" podID="c7550496-b283-48c0-ad55-2d426a0c9e7c" containerID="56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a" exitCode=0 Jul 31 21:28:04.293728 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.293692 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" Jul 31 21:28:04.293946 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.293728 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" event={"ID":"c7550496-b283-48c0-ad55-2d426a0c9e7c","Type":"ContainerDied","Data":"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a"} Jul 31 21:28:04.293946 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.293768 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br" event={"ID":"c7550496-b283-48c0-ad55-2d426a0c9e7c","Type":"ContainerDied","Data":"2d561075646a15d634700f478acc029440c2a51ce2459aedc0ebd4556e014ac7"} Jul 31 21:28:04.293946 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.293787 2544 scope.go:117] "RemoveContainer" containerID="56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a" Jul 31 21:28:04.302744 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.302724 2544 scope.go:117] "RemoveContainer" containerID="56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a" Jul 31 21:28:04.303139 ip-10-0-141-177 kubenswrapper[2544]: E0731 21:28:04.303120 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a\": container with ID starting with 56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a not found: ID does not exist" containerID="56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a" Jul 31 21:28:04.303180 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.303148 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a"} err="failed to get container status \"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a\": rpc error: code = NotFound desc = could not find container \"56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a\": container with ID starting with 56d1f6029cc1dc950f3acadbaf693a4b1d00e7e0477192b726776b2b0678728a not found: ID does not exist" Jul 31 21:28:04.311449 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.311415 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:28:04.314910 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:04.314884 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b4cb74/ipp-route-b4cb74-kserve-777cb7696c-xr8br"] Jul 31 21:28:06.113828 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:28:06.113793 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7550496-b283-48c0-ad55-2d426a0c9e7c" path="/var/lib/kubelet/pods/c7550496-b283-48c0-ad55-2d426a0c9e7c/volumes" Jul 31 21:29:24.543198 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.543162 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:29:24.543679 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.543420 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" podUID="910463bc-3500-439c-81d3-3423cdb2cdc9" containerName="maas-api" containerID="cri-o://0ccd2e2ae809e1a052b41f093ccc0376876e5807c36b4d5cb8a6983d00e94cab" gracePeriod=30 Jul 31 21:29:24.705681 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.705643 2544 generic.go:356] "Generic (PLEG): container finished" podID="910463bc-3500-439c-81d3-3423cdb2cdc9" containerID="0ccd2e2ae809e1a052b41f093ccc0376876e5807c36b4d5cb8a6983d00e94cab" exitCode=0 Jul 31 21:29:24.705853 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.705717 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" event={"ID":"910463bc-3500-439c-81d3-3423cdb2cdc9","Type":"ContainerDied","Data":"0ccd2e2ae809e1a052b41f093ccc0376876e5807c36b4d5cb8a6983d00e94cab"} Jul 31 21:29:24.802036 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.801975 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:29:24.979491 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.979449 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvlwn\" (UniqueName: \"kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn\") pod \"910463bc-3500-439c-81d3-3423cdb2cdc9\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " Jul 31 21:29:24.979697 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.979499 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca\") pod \"910463bc-3500-439c-81d3-3423cdb2cdc9\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " Jul 31 21:29:24.979697 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.979539 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls\") pod \"910463bc-3500-439c-81d3-3423cdb2cdc9\" (UID: \"910463bc-3500-439c-81d3-3423cdb2cdc9\") " Jul 31 21:29:24.979910 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.979862 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "910463bc-3500-439c-81d3-3423cdb2cdc9" (UID: "910463bc-3500-439c-81d3-3423cdb2cdc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 21:29:24.981825 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.981798 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "910463bc-3500-439c-81d3-3423cdb2cdc9" (UID: "910463bc-3500-439c-81d3-3423cdb2cdc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 21:29:24.981925 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:24.981880 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn" (OuterVolumeSpecName: "kube-api-access-vvlwn") pod "910463bc-3500-439c-81d3-3423cdb2cdc9" (UID: "910463bc-3500-439c-81d3-3423cdb2cdc9"). InnerVolumeSpecName "kube-api-access-vvlwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 21:29:25.080640 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.080538 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlwn\" (UniqueName: \"kubernetes.io/projected/910463bc-3500-439c-81d3-3423cdb2cdc9-kube-api-access-vvlwn\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:29:25.080640 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.080576 2544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/910463bc-3500-439c-81d3-3423cdb2cdc9-openshift-service-ca\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:29:25.080640 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.080588 2544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/910463bc-3500-439c-81d3-3423cdb2cdc9-maas-api-tls\") on node \"ip-10-0-141-177.ec2.internal\" DevicePath \"\"" Jul 31 21:29:25.712860 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.712824 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" Jul 31 21:29:25.712860 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.712839 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n" event={"ID":"910463bc-3500-439c-81d3-3423cdb2cdc9","Type":"ContainerDied","Data":"074c06d3b75c5766cd4dc4bb76a748b714889cd21903434a059da47fea01c498"} Jul 31 21:29:25.713412 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.712937 2544 scope.go:117] "RemoveContainer" containerID="0ccd2e2ae809e1a052b41f093ccc0376876e5807c36b4d5cb8a6983d00e94cab" Jul 31 21:29:25.734296 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.734263 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:29:25.739755 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:25.739729 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c3fd2-79dbcf4474-pbz5n"] Jul 31 21:29:26.112939 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:26.112908 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910463bc-3500-439c-81d3-3423cdb2cdc9" path="/var/lib/kubelet/pods/910463bc-3500-439c-81d3-3423cdb2cdc9/volumes" Jul 31 21:29:26.986393 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:26.986356 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:29:44.689223 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:29:44.689183 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:30:03.831412 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:03.831370 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx8cg"] Jul 31 21:30:10.831289 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:10.831249 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7564444965-4pjl8_6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98/authorino/0.log" Jul 31 21:30:17.155050 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:17.154963 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-786d856849-zw2k2_e0035325-c47c-4bbd-9a7c-17636046bf60/copy-manifests/0.log" Jul 31 21:30:17.161890 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:17.161850 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-786d856849-zw2k2_e0035325-c47c-4bbd-9a7c-17636046bf60/manager/0.log" Jul 31 21:30:18.755795 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:18.755759 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7564444965-4pjl8_6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98/authorino/0.log" Jul 31 21:30:19.127591 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:19.127558 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-qpmg9_59aca7dd-abfd-47f7-8f52-d51f650b2617/kuadrant-console-plugin/0.log" Jul 31 21:30:19.369515 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:19.369480 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dx8cg_66105734-628b-47f1-b73c-e1d420d3a6bb/limitador/0.log" Jul 31 21:30:20.078060 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:20.078026 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d469b876d-2v26d_8b7fad3e-521b-4699-bbf8-b8d5b29dabeb/kube-auth-proxy/0.log" Jul 31 21:30:21.030991 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:21.030959 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-j687v_b2cedb60-722e-443b-80e5-9b98bde9f3b0/main/0.log" Jul 31 21:30:21.375438 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:21.375402 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr6sqf_89d205be-3a52-496d-af8d-6e238cc57448/main/0.log" Jul 31 21:30:30.380566 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:30.380530 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b6gsd_99e6c527-9518-4a36-81c2-ea2634d4ecca/global-pull-secret-syncer/0.log" Jul 31 21:30:30.766447 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:30.766365 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x89qb_9027ab96-d2cc-474c-b92d-9865ba493785/konnectivity-agent/0.log" Jul 31 21:30:30.897978 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:30.897944 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-177.ec2.internal_9ceb4d3aec329af764e1753333df00b0/haproxy/0.log" Jul 31 21:30:35.509232 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:35.509187 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7564444965-4pjl8_6dbf9971-3409-44d2-9e9a-cb8e2e2e3b98/authorino/0.log" Jul 31 21:30:35.733802 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:35.733772 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-qpmg9_59aca7dd-abfd-47f7-8f52-d51f650b2617/kuadrant-console-plugin/0.log" Jul 31 21:30:35.928602 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:35.928571 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dx8cg_66105734-628b-47f1-b73c-e1d420d3a6bb/limitador/0.log" Jul 31 21:30:37.710672 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.710647 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/alertmanager/0.log" Jul 31 21:30:37.740258 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.740221 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/config-reloader/0.log" Jul 31 21:30:37.768096 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.768022 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/kube-rbac-proxy-web/0.log" Jul 31 21:30:37.804761 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.804734 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/kube-rbac-proxy/0.log" Jul 31 21:30:37.835478 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.835448 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/kube-rbac-proxy-metric/0.log" Jul 31 21:30:37.862579 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.862552 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/prom-label-proxy/0.log" Jul 31 21:30:37.890360 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.890336 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d28cabc-587e-49e7-9a8d-1c92f246be85/init-config-reloader/0.log" Jul 31 21:30:37.969729 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.969699 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-hhprh_a9a01394-3275-4296-baee-b1ac06a37452/kube-state-metrics/0.log" Jul 31 21:30:37.998328 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:37.998296 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-hhprh_a9a01394-3275-4296-baee-b1ac06a37452/kube-rbac-proxy-main/0.log" Jul 31 21:30:38.020929 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.020823 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-hhprh_a9a01394-3275-4296-baee-b1ac06a37452/kube-rbac-proxy-self/0.log" Jul 31 21:30:38.048666 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.048634 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d8986d654-f46p4_af20a449-b019-4665-8709-039623fefc2b/metrics-server/0.log" Jul 31 21:30:38.074774 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.074749 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-7rtj2_be14b32c-3b96-4155-ae28-b1237fec6e63/monitoring-plugin/0.log" Jul 31 21:30:38.104771 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.104747 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh2qj_3a4c2b8b-926d-416c-bbc3-5af77737ed39/node-exporter/0.log" Jul 31 21:30:38.128203 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.128157 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh2qj_3a4c2b8b-926d-416c-bbc3-5af77737ed39/kube-rbac-proxy/0.log" Jul 31 21:30:38.152104 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.152076 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh2qj_3a4c2b8b-926d-416c-bbc3-5af77737ed39/init-textfile/0.log" Jul 31 21:30:38.393600 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.393566 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-knvzx_3dce5282-17a5-497e-a8dc-15e9c8cb8800/kube-rbac-proxy-main/0.log" Jul 31 21:30:38.414313 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.414285 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-knvzx_3dce5282-17a5-497e-a8dc-15e9c8cb8800/kube-rbac-proxy-self/0.log" Jul 31 21:30:38.462199 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.462174 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-knvzx_3dce5282-17a5-497e-a8dc-15e9c8cb8800/openshift-state-metrics/0.log" Jul 31 21:30:38.526762 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.526736 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/prometheus/0.log" Jul 31 21:30:38.550737 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.550665 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/config-reloader/0.log" Jul 31 21:30:38.582669 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.582640 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/thanos-sidecar/0.log" Jul 31 21:30:38.607562 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.607532 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/kube-rbac-proxy-web/0.log" Jul 31 21:30:38.635765 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.635736 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/kube-rbac-proxy/0.log" Jul 31 21:30:38.670991 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.670965 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/kube-rbac-proxy-thanos/0.log" Jul 31 21:30:38.700637 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.700612 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c10e4b4-6373-4ee6-8bbb-e3e817630ada/init-config-reloader/0.log" Jul 31 21:30:38.731765 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.731734 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-d9rwz_01616c7a-ffbe-42da-89ea-b24e372363bc/prometheus-operator/0.log" Jul 31 21:30:38.758232 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.758213 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-d9rwz_01616c7a-ffbe-42da-89ea-b24e372363bc/kube-rbac-proxy/0.log" Jul 31 21:30:38.792900 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.792855 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-s27h4_691be1d0-d323-4627-b52a-03af75a442aa/prometheus-operator-admission-webhook/0.log" Jul 31 21:30:38.828782 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.828709 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7489846744-5jshv_ed94aae4-944b-41b3-ada6-9cc6a78ff4e1/telemeter-client/0.log" Jul 31 21:30:38.852725 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.852702 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7489846744-5jshv_ed94aae4-944b-41b3-ada6-9cc6a78ff4e1/reload/0.log" Jul 31 21:30:38.876235 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.876209 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7489846744-5jshv_ed94aae4-944b-41b3-ada6-9cc6a78ff4e1/kube-rbac-proxy/0.log" Jul 31 21:30:38.908467 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.908434 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/thanos-query/0.log" Jul 31 21:30:38.935575 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.935542 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/kube-rbac-proxy-web/0.log" Jul 31 21:30:38.978254 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:38.978232 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/kube-rbac-proxy/0.log" Jul 31 21:30:39.002241 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:39.002210 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/prom-label-proxy/0.log" Jul 31 21:30:39.027718 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:39.027684 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/kube-rbac-proxy-rules/0.log" Jul 31 21:30:39.072105 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:39.072074 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f5d9b6d48-4kfg7_1f62f997-53ea-474f-beaa-5244f0d1470d/kube-rbac-proxy-metrics/0.log" Jul 31 21:30:40.602127 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602091 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86"] Jul 31 21:30:40.602507 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602477 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910463bc-3500-439c-81d3-3423cdb2cdc9" containerName="maas-api" Jul 31 21:30:40.602507 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602490 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="910463bc-3500-439c-81d3-3423cdb2cdc9" containerName="maas-api" Jul 31 21:30:40.602507 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602498 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7550496-b283-48c0-ad55-2d426a0c9e7c" containerName="main" Jul 31 21:30:40.602507 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602503 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7550496-b283-48c0-ad55-2d426a0c9e7c" containerName="main" Jul 31 21:30:40.602643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602520 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db85565c-4f7f-4265-a86f-371e0cf8a33d" containerName="maas-api" Jul 31 21:30:40.602643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602526 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="db85565c-4f7f-4265-a86f-371e0cf8a33d" containerName="maas-api" Jul 31 21:30:40.602643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602590 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="db85565c-4f7f-4265-a86f-371e0cf8a33d" containerName="maas-api" Jul 31 21:30:40.602643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602597 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="910463bc-3500-439c-81d3-3423cdb2cdc9" containerName="maas-api" Jul 31 21:30:40.602643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.602604 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7550496-b283-48c0-ad55-2d426a0c9e7c" containerName="main" Jul 31 21:30:40.605711 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.605695 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.608843 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.608822 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpvlq\"/\"openshift-service-ca.crt\"" Jul 31 21:30:40.608983 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.608848 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hpvlq\"/\"default-dockercfg-869gk\"" Jul 31 21:30:40.608983 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.608823 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpvlq\"/\"kube-root-ca.crt\"" Jul 31 21:30:40.614493 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.614466 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86"] Jul 31 21:30:40.700696 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.700662 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-sys\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.700901 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.700722 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-lib-modules\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.700901 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.700821 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ptq\" (UniqueName: \"kubernetes.io/projected/9360d779-daf6-4793-9f78-f6c615009d7a-kube-api-access-l7ptq\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.700901 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.700895 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-podres\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.701017 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.700941 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-proc\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802083 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802047 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-lib-modules\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802256 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802106 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ptq\" (UniqueName: \"kubernetes.io/projected/9360d779-daf6-4793-9f78-f6c615009d7a-kube-api-access-l7ptq\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802256 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802148 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-podres\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802256 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802196 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-proc\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802256 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802225 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-sys\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802256 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802241 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-lib-modules\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802276 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-proc\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802292 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-podres\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.802484 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.802365 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9360d779-daf6-4793-9f78-f6c615009d7a-sys\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.810688 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.810649 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ptq\" (UniqueName: \"kubernetes.io/projected/9360d779-daf6-4793-9f78-f6c615009d7a-kube-api-access-l7ptq\") pod \"perf-node-gather-daemonset-w5c86\" (UID: \"9360d779-daf6-4793-9f78-f6c615009d7a\") " pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:40.916668 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:40.916578 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:41.059081 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:41.059055 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86"] Jul 31 21:30:41.061243 ip-10-0-141-177 kubenswrapper[2544]: W0731 21:30:41.061211 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9360d779_daf6_4793_9f78_f6c615009d7a.slice/crio-d5e29b8a07c69b3d4ba0d8499e2982c9b6aaf0764a323d30f3358033fae61e57 WatchSource:0}: Error finding container d5e29b8a07c69b3d4ba0d8499e2982c9b6aaf0764a323d30f3358033fae61e57: Status 404 returned error can't find the container with id d5e29b8a07c69b3d4ba0d8499e2982c9b6aaf0764a323d30f3358033fae61e57 Jul 31 21:30:41.076713 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:41.076676 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" event={"ID":"9360d779-daf6-4793-9f78-f6c615009d7a","Type":"ContainerStarted","Data":"d5e29b8a07c69b3d4ba0d8499e2982c9b6aaf0764a323d30f3358033fae61e57"} Jul 31 21:30:41.608305 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:41.608270 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-fx8pm_836b7c03-2d71-4d64-a018-a2f319e82592/download-server/0.log" Jul 31 21:30:42.083970 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:42.083841 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" event={"ID":"9360d779-daf6-4793-9f78-f6c615009d7a","Type":"ContainerStarted","Data":"7c0156d029fd1c73219b9a781bef6dcce1d1943fd663c7d6d43de7c99b69bb52"} Jul 31 21:30:42.084124 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:42.084018 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:42.100241 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:42.100193 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" podStartSLOduration=2.10017655 podStartE2EDuration="2.10017655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 21:30:42.09927884 +0000 UTC m=+3620.475652019" watchObservedRunningTime="2026-07-31 21:30:42.10017655 +0000 UTC m=+3620.476549732" Jul 31 21:30:43.260422 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:43.260392 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fdttp_301dda6b-10e8-4f0e-be29-f4f520f6a061/dns/0.log" Jul 31 21:30:43.281876 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:43.281831 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fdttp_301dda6b-10e8-4f0e-be29-f4f520f6a061/kube-rbac-proxy/0.log" Jul 31 21:30:43.453156 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:43.453127 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8hvl_c2ab9526-3efa-4a3c-82aa-b57b0b56d2f3/dns-node-resolver/0.log" Jul 31 21:30:44.015643 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:44.015615 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-68whh_585a0b01-8e0b-4f08-a98e-33508e42d0bd/node-ca/0.log" Jul 31 21:30:45.171652 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:45.171622 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d469b876d-2v26d_8b7fad3e-521b-4699-bbf8-b8d5b29dabeb/kube-auth-proxy/0.log" Jul 31 21:30:46.136967 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:46.136941 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gchrn_0ac1a04c-44bc-4eb3-aefe-aae6f3c64f40/serve-healthcheck-canary/0.log" Jul 31 21:30:46.710892 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:46.710841 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxz27_cb8484d7-b167-466e-8a4f-1c8c0a85534d/kube-rbac-proxy/0.log" Jul 31 21:30:46.731235 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:46.731209 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxz27_cb8484d7-b167-466e-8a4f-1c8c0a85534d/exporter/0.log" Jul 31 21:30:46.751159 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:46.751129 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jxz27_cb8484d7-b167-466e-8a4f-1c8c0a85534d/extractor/0.log" Jul 31 21:30:48.099661 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:48.099630 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hpvlq/perf-node-gather-daemonset-w5c86" Jul 31 21:30:50.123340 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:50.123307 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-786d856849-zw2k2_e0035325-c47c-4bbd-9a7c-17636046bf60/manager/0.log" Jul 31 21:30:50.145220 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:50.145182 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-786d856849-zw2k2_e0035325-c47c-4bbd-9a7c-17636046bf60/copy-manifests/0.log" Jul 31 21:30:51.434860 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:51.434827 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-595ffc7687-79dp7_e97f1994-18ef-42b8-ba3c-65ad09b7d720/manager/0.log" Jul 31 21:30:51.589637 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:51.589600 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d469b876d-2v26d_8b7fad3e-521b-4699-bbf8-b8d5b29dabeb/kube-auth-proxy/0.log" Jul 31 21:30:53.341298 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:53.341268 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d469b876d-2v26d_8b7fad3e-521b-4699-bbf8-b8d5b29dabeb/kube-auth-proxy/0.log" Jul 31 21:30:57.892983 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:57.892948 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2tm82_2dda82ab-5555-4f13-b159-df10a22d5b36/kube-multus/0.log" Jul 31 21:30:58.076077 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.076050 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/kube-multus-additional-cni-plugins/0.log" Jul 31 21:30:58.098716 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.098686 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/egress-router-binary-copy/0.log" Jul 31 21:30:58.122705 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.122677 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/cni-plugins/0.log" Jul 31 21:30:58.145067 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.144993 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/bond-cni-plugin/0.log" Jul 31 21:30:58.169596 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.169567 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/routeoverride-cni/0.log" Jul 31 21:30:58.194102 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.194075 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/whereabouts-cni-bincopy/0.log" Jul 31 21:30:58.217268 ip-10-0-141-177 kubenswrapper[2544]: I0731 21:30:58.217232 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8cfg_f92dc492-6985-4129-817c-60cbd61a0d6e/whereabouts-cni/0.log"