Jul 31 18:44:10.215494 ip-10-0-139-43 systemd[1]: Starting Kubernetes Kubelet... Jul 31 18:44:10.656841 ip-10-0-139-43 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:44:10.656841 ip-10-0-139-43 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 18:44:10.656841 ip-10-0-139-43 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:44:10.656841 ip-10-0-139-43 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 18:44:10.656841 ip-10-0-139-43 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 18:44:10.658697 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.658610 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 18:44:10.662088 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662072 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.662088 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662089 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662093 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662097 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662100 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662103 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662106 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662109 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662112 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662115 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662118 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662120 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662123 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662125 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662128 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662134 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662137 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662139 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662141 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662144 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662146 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.662153 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662149 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662151 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662154 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662157 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662159 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662162 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662165 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662168 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662170 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662173 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662175 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662191 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662195 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662197 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662200 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662202 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662205 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662208 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662210 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662212 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.662665 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662215 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662219 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662223 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662226 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662228 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662231 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662234 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662237 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662240 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662242 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662245 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662247 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662250 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662252 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662256 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662260 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662264 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662267 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662269 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.663158 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662272 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662276 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662278 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662281 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662284 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662286 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662288 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662291 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662293 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662296 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662298 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662301 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662303 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662306 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662308 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662311 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662313 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662316 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662319 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662323 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.663654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662326 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662329 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662332 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662335 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662337 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662340 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662343 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662347 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662349 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662352 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662354 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662357 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662359 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662361 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662364 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662367 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662369 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662372 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.662374 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.664167 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663070 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663077 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663082 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663087 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663091 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663095 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663098 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663101 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663104 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663107 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663109 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663112 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663114 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663117 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663120 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663123 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663125 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663128 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663130 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663133 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.664686 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663136 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663139 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663141 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663144 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663146 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663149 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663151 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663153 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663156 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663159 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663161 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663164 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663167 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663169 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663172 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663174 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663177 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663197 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663199 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663202 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.665200 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663204 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663207 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663210 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663213 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663216 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663219 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663222 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663226 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663229 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663232 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663234 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663237 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663240 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663243 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663246 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663250 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663253 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663256 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663259 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.665716 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663262 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663265 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663267 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663270 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663273 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663276 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663279 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663282 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663285 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663287 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663290 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663292 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663295 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663298 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663302 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663304 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663307 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663309 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663312 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663314 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.666174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663317 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663320 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663323 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663325 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663328 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663330 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663333 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663335 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663338 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663340 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663343 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663345 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663347 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663350 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663352 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663355 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663357 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663360 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663363 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.666679 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663365 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663722 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663726 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663730 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663732 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663736 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663740 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663743 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663746 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663750 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663752 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663755 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663758 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663760 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663764 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663766 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663768 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663770 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663773 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.667156 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663776 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663779 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663781 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663784 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663786 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663788 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663790 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663793 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663795 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663798 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663800 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663805 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663808 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663810 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663813 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663815 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663818 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663820 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663823 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663825 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.667616 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663827 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663830 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663832 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663835 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663837 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663840 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663842 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663845 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663848 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663850 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663853 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663855 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663858 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663860 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663863 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663865 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663867 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663870 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663872 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663874 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.668112 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663877 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663880 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663882 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663886 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663889 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663891 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663893 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663896 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663899 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663901 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663904 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663907 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663911 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663914 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663917 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663920 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663923 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663925 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663928 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.668599 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663930 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663932 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663935 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663938 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663940 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663943 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663945 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663947 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663950 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663952 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663954 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663957 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663959 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663962 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663964 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663966 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663970 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663972 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663975 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663977 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.669068 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663979 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.663982 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664056 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664067 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664074 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664078 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664082 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664085 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664091 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664095 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664099 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664101 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664105 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664108 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664111 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664115 2566 flags.go:64] FLAG: --cgroup-root="" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664118 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664121 2566 flags.go:64] FLAG: --client-ca-file="" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664124 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664126 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664130 2566 flags.go:64] FLAG: --cluster-domain="" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664132 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664135 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664138 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 18:44:10.669564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664141 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664145 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664148 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664151 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664155 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664158 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664161 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664164 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664167 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664170 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664175 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664191 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664195 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664198 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664201 2566 flags.go:64] FLAG: --enable-server="true" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664204 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664209 2566 flags.go:64] FLAG: --event-burst="100" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664213 2566 flags.go:64] FLAG: --event-qps="50" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664216 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664219 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664222 2566 flags.go:64] FLAG: --eviction-hard="" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664226 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664228 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664231 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664235 2566 flags.go:64] FLAG: --eviction-soft="" Jul 31 18:44:10.670135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664238 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664241 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664244 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664246 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664249 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664252 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664255 2566 flags.go:64] FLAG: --feature-gates="" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664258 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664261 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664264 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664267 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664271 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664274 2566 flags.go:64] FLAG: --help="false" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664276 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-139-43.ec2.internal" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664279 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664282 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664284 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664288 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664291 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664294 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664296 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664299 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664301 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 18:44:10.670747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664306 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664309 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664311 2566 flags.go:64] FLAG: --kube-reserved="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664314 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664317 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664320 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664323 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664325 2566 flags.go:64] FLAG: --lock-file="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664328 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664331 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664334 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664339 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664342 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664344 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664347 2566 flags.go:64] FLAG: --logging-format="text" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664349 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664353 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664355 2566 flags.go:64] FLAG: --manifest-url="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664358 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664362 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664366 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664370 2566 flags.go:64] FLAG: --max-pods="110" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664372 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664375 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664378 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 18:44:10.671379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664381 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664384 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664387 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664390 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664396 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664399 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664402 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664406 2566 flags.go:64] FLAG: --pod-cidr="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664409 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664413 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664416 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664419 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664422 2566 flags.go:64] FLAG: --port="10250" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664425 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664427 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01186dd8104f949f9" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664430 2566 flags.go:64] FLAG: --qos-reserved="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664433 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664436 2566 flags.go:64] FLAG: --register-node="true" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664439 2566 flags.go:64] FLAG: --register-with-taints="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664443 2566 flags.go:64] FLAG: --registry-burst="10" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664445 2566 flags.go:64] FLAG: --registry-qps="5" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664448 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664451 2566 flags.go:64] FLAG: --reserved-memory="" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664454 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664457 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 18:44:10.672336 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664460 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664463 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664465 2566 flags.go:64] FLAG: --runonce="false" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664470 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664473 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664476 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664479 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664481 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664484 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664487 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664490 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664493 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664496 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664498 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664502 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664506 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664508 2566 flags.go:64] FLAG: --system-cgroups="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664511 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664516 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664519 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664522 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664525 2566 flags.go:64] FLAG: --tls-min-version="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664528 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664531 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664533 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 18:44:10.673122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664536 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664539 2566 flags.go:64] FLAG: --v="2" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664543 2566 flags.go:64] FLAG: --version="false" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664547 2566 flags.go:64] FLAG: --vmodule="" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664550 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.664553 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664674 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664677 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664681 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664683 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664689 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664692 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664695 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664698 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664701 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664704 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664706 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664709 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664711 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664714 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664716 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.673795 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664720 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664723 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664727 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664731 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664733 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664736 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664739 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664742 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664744 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664746 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664749 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664751 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664756 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664759 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664762 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664764 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664767 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664769 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664771 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.674333 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664774 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664777 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664780 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664782 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664785 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664787 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664789 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664792 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664794 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664796 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664799 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664801 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664803 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664808 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664810 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664813 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664815 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664818 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664821 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664823 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.674808 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664826 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664828 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664830 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664832 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664835 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664837 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664840 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664842 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664845 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664847 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664850 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664852 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664855 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664857 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664860 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664862 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664865 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664867 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664869 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664872 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.675338 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664874 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664877 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664879 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664883 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664886 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664891 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664894 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664897 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664899 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664902 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664905 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664907 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664909 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664912 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664915 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664917 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664921 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664924 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664926 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.675814 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664929 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664932 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664934 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664936 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664939 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.664941 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.665780 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.674121 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.674137 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.675024 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.675036 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675096 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675102 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675105 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675108 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.676293 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675111 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675114 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675116 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675119 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675121 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675124 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675127 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675129 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675132 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675135 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675137 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675140 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675142 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675145 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675147 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675150 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675153 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675156 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675158 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675161 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.676699 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675163 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675167 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675171 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675174 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675188 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675192 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675194 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675197 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675200 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675203 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675205 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675208 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675211 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675213 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675216 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675219 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675221 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675224 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675227 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.677204 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675229 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675231 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675234 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675237 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675240 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675242 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675245 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675247 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675249 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675252 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675254 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675257 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675259 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675263 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675266 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675269 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675272 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675275 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675278 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.677654 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675281 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675283 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675286 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675289 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675291 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675294 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675297 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675299 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675301 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675304 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675306 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675309 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675312 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675314 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675316 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675319 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675321 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675325 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675327 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675330 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.678155 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675332 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675334 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675337 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675339 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675342 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675344 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675347 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675349 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675352 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675354 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675357 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675360 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675363 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675365 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675368 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675371 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.678748 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675375 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.675380 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675519 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675525 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675528 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675531 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675534 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675536 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675539 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675542 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675544 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675546 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675549 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675551 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675555 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 18:44:10.679145 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675557 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675560 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675563 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675565 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675568 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675570 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675573 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675575 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675578 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675580 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675582 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675585 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675588 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675590 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675593 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675595 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675598 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675602 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675605 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675608 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 18:44:10.679669 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675611 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675614 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675618 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675621 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675624 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675626 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675629 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675632 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675635 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675637 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675640 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675642 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675645 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675648 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675650 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675652 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675654 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675657 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675659 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 18:44:10.680165 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675662 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675665 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675668 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675671 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675673 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675676 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675678 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675681 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675684 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675686 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675688 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675691 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675693 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675695 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675698 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675700 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675702 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675705 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675707 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 18:44:10.680637 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675709 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675712 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675714 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675717 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675719 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675722 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675724 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675728 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675730 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675733 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675735 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675737 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675740 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675742 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675744 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675747 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675750 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675752 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675755 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675757 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 18:44:10.681093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675759 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675762 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675764 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675767 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675769 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675771 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675774 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:10.675776 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.675781 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.675888 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.678654 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.680070 2566 server.go:1019] "Starting client certificate rotation" Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.680171 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 18:44:10.681601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.680228 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 18:44:10.683077 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.683064 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 18:44:10.684307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.684276 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 18:44:10.699847 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.699828 2566 log.go:25] "Validated CRI v1 runtime API" Jul 31 18:44:10.701891 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.701878 2566 log.go:25] "Validated CRI v1 image API" Jul 31 18:44:10.703052 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.703035 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 18:44:10.705738 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.705717 2566 fs.go:135] Filesystem UUIDs: map[267afb26-ad5d-4607-aaef-1a34e4811546:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e0b98428-bf50-4b46-9ec8-1673b18830e8:/dev/nvme0n1p3] Jul 31 18:44:10.705821 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.705743 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 18:44:10.711048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.711024 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 18:44:10.711944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.711832 2566 manager.go:216] Machine: {Timestamp:2026-07-31 18:44:10.710821926 +0000 UTC m=+0.368357005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105421 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2662bdeb187da9f2c9ec2e4b7e066b SystemUUID:ec2662bd-eb18-7da9-f2c9-ec2e4b7e066b BootID:451afe69-f1de-4c54-8a99-4c0e09a98ad9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:f0:62:84:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:f0:62:84:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:97:91:a6:93:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 18:44:10.711944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.711936 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 18:44:10.712048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.712012 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 18:44:10.713230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713200 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 18:44:10.713369 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713232 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 18:44:10.713967 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713958 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 18:44:10.714000 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713970 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 18:44:10.714000 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713978 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 18:44:10.714000 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.713989 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 18:44:10.715339 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.715328 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 18:44:10.718899 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.718888 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 31 18:44:10.719021 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719010 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 18:44:10.719064 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719042 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 31 18:44:10.719064 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719052 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 18:44:10.719126 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719065 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 18:44:10.719126 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719073 2566 kubelet.go:398] "Adding apiserver pod source" Jul 31 18:44:10.719126 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719082 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 18:44:10.719452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719433 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 18:44:10.719528 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.719458 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 18:44:10.722093 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.722075 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 18:44:10.723742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.723729 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 18:44:10.723791 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.723759 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 18:44:10.724395 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724384 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 18:44:10.724452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724400 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 18:44:10.724452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724407 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 18:44:10.724452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724412 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 18:44:10.724452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724420 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 18:44:10.724452 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724425 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724530 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724536 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724543 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724548 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724557 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 18:44:10.724577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.724564 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 18:44:10.725279 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.725270 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 18:44:10.725279 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.725279 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 18:44:10.728489 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.728476 2566 server.go:1268] "Started kubelet" Jul 31 18:44:10.728612 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.728555 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 18:44:10.728671 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.728641 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 18:44:10.728716 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.728589 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 18:44:10.729507 ip-10-0-139-43 systemd[1]: Started Kubernetes Kubelet. Jul 31 18:44:10.729857 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.729837 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 18:44:10.730135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.730079 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 31 18:44:10.732643 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.732626 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p4hqx" Jul 31 18:44:10.733064 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.733048 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 18:44:10.733579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.733563 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 18:44:10.733579 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.733565 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 18:44:10.733716 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.733601 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 18:44:10.734718 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.733675 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-43.ec2.internal.18c7741e4acc83d7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-43.ec2.internal,UID:ip-10-0-139-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-43.ec2.internal,},FirstTimestamp:2026-07-31 18:44:10.728448983 +0000 UTC m=+0.385984051,LastTimestamp:2026-07-31 18:44:10.728448983 +0000 UTC m=+0.385984051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-43.ec2.internal,}" Jul 31 18:44:10.737701 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.737686 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 18:44:10.737762 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.737700 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 18:44:10.737762 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.737749 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:10.738454 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738271 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 18:44:10.738454 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738292 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 18:44:10.738454 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738393 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 31 18:44:10.738454 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738403 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 31 18:44:10.738686 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738509 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 18:44:10.738811 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738794 2566 factory.go:153] Registering CRI-O factory Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738839 2566 factory.go:223] Registration of the crio container factory successfully Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738890 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738897 2566 factory.go:55] Registering systemd factory Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738903 2566 factory.go:223] Registration of the systemd container factory successfully Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738918 2566 factory.go:103] Registering Raw factory Jul 31 18:44:10.739028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.738928 2566 manager.go:1196] Started watching for new ooms in manager Jul 31 18:44:10.739367 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.739352 2566 manager.go:349] Starting recovery of all containers Jul 31 18:44:10.739431 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.739388 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 18:44:10.741357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.741332 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p4hqx" Jul 31 18:44:10.751356 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.751203 2566 manager.go:354] Recovery completed Jul 31 18:44:10.752412 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.752394 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:10.756540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.756528 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:10.758423 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758406 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:10.758502 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758435 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:10.758502 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758446 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:10.758958 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758945 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 18:44:10.759013 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758960 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 18:44:10.759013 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758978 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 31 18:44:10.759013 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.758981 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:10.759636 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.759623 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:10.759691 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.759648 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:10.759691 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.759661 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:10.760287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.760275 2566 policy_none.go:49] "None policy: Start" Jul 31 18:44:10.760326 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.760296 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 18:44:10.760326 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.760308 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 18:44:10.761346 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.761328 2566 policy_none.go:47] "Start" Jul 31 18:44:10.812929 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.812903 2566 manager.go:337] "Starting Device Plugin manager" Jul 31 18:44:10.813052 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.812954 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 18:44:10.813052 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.812969 2566 server.go:87] "Starting device plugin registration server" Jul 31 18:44:10.813310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.813297 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 18:44:10.813357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.813315 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 18:44:10.813427 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.813406 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 18:44:10.813559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.813530 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 18:44:10.813559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.813544 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 18:44:10.814091 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.814069 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 18:44:10.814211 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.814129 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:10.907972 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.907878 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 18:44:10.909238 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.909220 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 18:44:10.909300 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.909261 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 18:44:10.909300 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.909286 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 18:44:10.909364 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.909322 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 18:44:10.912168 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.912149 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:10.914351 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.914336 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:10.915107 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.915093 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:10.915153 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.915125 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:10.915153 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.915140 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:10.915242 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.915163 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-43.ec2.internal" Jul 31 18:44:10.924319 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:10.924303 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-43.ec2.internal" Jul 31 18:44:10.924404 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.924327 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-43.ec2.internal\": node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:10.940815 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:10.940789 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:11.009463 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.009429 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal"] Jul 31 18:44:11.009588 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.009515 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:11.010968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.010954 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:11.011019 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.010982 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:11.011019 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.010994 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:11.012340 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.012328 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:11.012462 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.012447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.012505 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.012491 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:11.013145 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013126 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:11.013230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013159 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:11.013230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013171 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:11.013230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013126 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:11.013230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013210 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:11.013230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.013224 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:11.014514 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.014499 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.014571 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.014526 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 18:44:11.015234 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.015217 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientMemory" Jul 31 18:44:11.015313 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.015248 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 18:44:11.015313 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.015265 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeHasSufficientPID" Jul 31 18:44:11.028717 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:11.028697 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-43.ec2.internal\" not found" node="ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.032268 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:11.032252 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-43.ec2.internal\" not found" node="ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.039807 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.039790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.039878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.039816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.039878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.039832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63165082ff7f5713045b9ebffd02955c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-43.ec2.internal\" (UID: \"63165082ff7f5713045b9ebffd02955c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.041882 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:11.041867 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:11.140688 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.140798 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.140798 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63165082ff7f5713045b9ebffd02955c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-43.ec2.internal\" (UID: \"63165082ff7f5713045b9ebffd02955c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.140798 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140750 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63165082ff7f5713045b9ebffd02955c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-43.ec2.internal\" (UID: \"63165082ff7f5713045b9ebffd02955c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.140798 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140779 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.140917 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:11.140751 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/258b4f1f71dbda6e0fd0f662bb2ecc98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal\" (UID: \"258b4f1f71dbda6e0fd0f662bb2ecc98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:11.142764 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:11.142740 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:11.243635 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:11.243554 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.451130 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.451089 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:12.454198 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.454166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:12.463752 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.463733 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.564278 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.564226 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.664876 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.664793 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.765306 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.765274 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.798469 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.798435 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 18:44:12.799048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.798622 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 18:44:12.799048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.798638 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 18:44:12.848605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.848571 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:12.865792 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.865771 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:12.867926 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.867900 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 18:39:11 +0000 UTC" deadline="2028-03-23 01:43:19.007006096 +0000 UTC" Jul 31 18:44:12.867973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.867926 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14406h59m6.139082711s" Jul 31 18:44:12.958025 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:12.957993 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63165082ff7f5713045b9ebffd02955c.slice/crio-395a8ff3fb0fd93f448a2c069b4a2de82b5612e818d84eae1bb29a427e666342 WatchSource:0}: Error finding container 395a8ff3fb0fd93f448a2c069b4a2de82b5612e818d84eae1bb29a427e666342: Status 404 returned error can't find the container with id 395a8ff3fb0fd93f448a2c069b4a2de82b5612e818d84eae1bb29a427e666342 Jul 31 18:44:12.961573 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:12.961541 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258b4f1f71dbda6e0fd0f662bb2ecc98.slice/crio-7932ec3192097ee256abb9976181fc50f37a33213a812ed56b433011406c9081 WatchSource:0}: Error finding container 7932ec3192097ee256abb9976181fc50f37a33213a812ed56b433011406c9081: Status 404 returned error can't find the container with id 7932ec3192097ee256abb9976181fc50f37a33213a812ed56b433011406c9081 Jul 31 18:44:12.964410 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:12.964395 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:44:12.966113 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:12.966088 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:13.032559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.032503 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" event={"ID":"258b4f1f71dbda6e0fd0f662bb2ecc98","Type":"ContainerStarted","Data":"7932ec3192097ee256abb9976181fc50f37a33213a812ed56b433011406c9081"} Jul 31 18:44:13.033479 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.033461 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" event={"ID":"63165082ff7f5713045b9ebffd02955c","Type":"ContainerStarted","Data":"395a8ff3fb0fd93f448a2c069b4a2de82b5612e818d84eae1bb29a427e666342"} Jul 31 18:44:13.067023 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.066989 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:13.167539 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.167484 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-43.ec2.internal\" not found" Jul 31 18:44:13.170579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.170561 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:13.258234 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.258163 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" Jul 31 18:44:13.271556 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.271533 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 18:44:13.272570 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.272556 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" Jul 31 18:44:13.284569 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.284550 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 18:44:13.839464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.839413 2566 apiserver.go:52] "Watching apiserver" Jul 31 18:44:13.849623 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.849593 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 18:44:13.850934 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.850908 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal","openshift-network-diagnostics/network-check-target-rfnxc","openshift-network-operator/iptables-alerter-js2hr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq","openshift-cluster-node-tuning-operator/tuned-tl4cq","openshift-dns/node-resolver-7tcvq","openshift-image-registry/node-ca-zt6dh","openshift-multus/multus-additional-cni-plugins-lx6kw","openshift-multus/multus-gmpwb","openshift-multus/network-metrics-daemon-nswnl","openshift-ovn-kubernetes/ovnkube-node-7chl4","kube-system/global-pull-secret-syncer-6gr7q","kube-system/konnectivity-agent-whppk"] Jul 31 18:44:13.852614 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.852598 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.853827 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.853810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.853932 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.853876 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:13.855111 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.855008 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:13.855111 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.855078 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:13.855781 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.855763 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qn59\"" Jul 31 18:44:13.855957 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.855942 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.856104 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.856090 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.856587 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.856562 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.857963 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.857946 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 18:44:13.858756 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.858739 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.859576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.859557 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.859708 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.859557 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.859895 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.859880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.860177 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.860159 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mnl8l\"" Jul 31 18:44:13.860258 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.860161 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 18:44:13.861518 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.860857 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.861729 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.861583 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bt6x\"" Jul 31 18:44:13.861729 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.861583 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.861905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.861887 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 18:44:13.862130 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.862109 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:13.862445 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.862428 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.863516 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.863174 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wqnfg\"" Jul 31 18:44:13.863642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.863624 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:13.863970 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.863950 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 18:44:13.865077 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.865057 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 18:44:13.865406 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.865389 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 18:44:13.865886 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.865866 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.866012 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.865989 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866187 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-27jb9\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866381 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866648 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866680 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2jtr\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.866732 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 18:44:13.867116 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867037 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 18:44:13.867474 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867302 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.867474 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867334 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:13.867474 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867448 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wkq7c\"" Jul 31 18:44:13.867474 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.867440 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:13.868238 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867797 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 18:44:13.868238 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.867952 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.868990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.868964 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.869110 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-kubernetes\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-sys\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-lib-modules\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-system-cni-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.869265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-iptables-alerter-script\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.869265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869228 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-run\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869249 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-etc-tuned\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-dbus\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4ms4\" (UniqueName: \"kubernetes.io/projected/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-kube-api-access-g4ms4\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-socket-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869385 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-systemd\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869406 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-host\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.869455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cj4\" (UniqueName: \"kubernetes.io/projected/a67104f9-4e22-41f3-aa18-5f401c8503be-kube-api-access-t8cj4\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869481 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-registration-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869547 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mldd6\"" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869610 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869735 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-conf\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-cnibin\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869776 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-host-slash\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-device-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869827 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-modprobe-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.869904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-var-lib-kubelet\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869922 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kmx\" (UniqueName: \"kubernetes.io/projected/0856a836-c4e8-4363-bcb1-aaa671091730-kube-api-access-s9kmx\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869969 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996cb3b4-342a-44e6-8e35-60887aab2638-tmp-dir\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.869991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqn8w\" (UniqueName: \"kubernetes.io/projected/996cb3b4-342a-44e6-8e35-60887aab2638-kube-api-access-bqn8w\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-sys-fs\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-tmp\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/996cb3b4-342a-44e6-8e35-60887aab2638-hosts-file\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-os-release\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870180 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870206 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-kubelet-config\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4mm\" (UniqueName: \"kubernetes.io/projected/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kube-api-access-wc4mm\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.870453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.870248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysconfig\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.871817 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.871673 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 18:44:13.873335 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.873312 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-thmd4\"" Jul 31 18:44:13.873643 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.873625 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 18:44:13.873708 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.873653 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 18:44:13.873767 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.873731 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 18:44:13.873949 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.873930 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 18:44:13.874443 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.874425 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 18:44:13.879903 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.879350 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 18:44:13.899676 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.899652 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6wfk" Jul 31 18:44:13.907601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.907578 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6wfk" Jul 31 18:44:13.959404 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.959369 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 18:44:13.961841 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.961811 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:13.970973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.970946 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-cni-binary-copy\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.970984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-multus\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-hostroot\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971033 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe239cdd-811b-4875-92d6-21ecf3c41138-konnectivity-ca\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52js7\" (UniqueName: \"kubernetes.io/projected/5c44670c-587b-4929-8898-672eb5b2ad8b-kube-api-access-52js7\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-config\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-run\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-os-release\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-kubelet\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4ms4\" (UniqueName: \"kubernetes.io/projected/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-kube-api-access-g4ms4\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-run\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-systemd\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971279 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-systemd\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971306 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-conf-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.971405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971344 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/82292e4a-5143-4fd1-8750-b8502a4041d6-kube-api-access-m7rbf\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-slash\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-netns\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971598 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-registration-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971686 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-systemd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-registration-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-node-log\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-script-lib\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971896 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.971951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-daemon-config\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971961 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-bin\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.971986 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqn8w\" (UniqueName: \"kubernetes.io/projected/996cb3b4-342a-44e6-8e35-60887aab2638-kube-api-access-bqn8w\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972013 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe239cdd-811b-4875-92d6-21ecf3c41138-agent-certs\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-ovn\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-netd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/996cb3b4-342a-44e6-8e35-60887aab2638-hosts-file\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-os-release\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972156 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-os-release\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-log-socket\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4mm\" (UniqueName: \"kubernetes.io/projected/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kube-api-access-wc4mm\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972300 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-kubernetes\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.972329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/242f0489-8ae7-42e5-bb8b-77d84a5fa711-serviceca\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-socket-dir-parent\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972374 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/996cb3b4-342a-44e6-8e35-60887aab2638-hosts-file\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-kubelet\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-var-lib-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-bin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-iptables-alerter-script\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-etc-tuned\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-dbus\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/242f0489-8ae7-42e5-bb8b-77d84a5fa711-host\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972659 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-socket-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-host\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972772 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cj4\" (UniqueName: \"kubernetes.io/projected/a67104f9-4e22-41f3-aa18-5f401c8503be-kube-api-access-t8cj4\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.972819 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972849 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-host\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-etc-kubernetes\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-systemd-units\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mg4v\" (UniqueName: \"kubernetes.io/projected/e6c8a0fa-aca4-466b-a901-7e785f569215-kube-api-access-9mg4v\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972941 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-kubernetes\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.972981 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-conf\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973021 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-cnibin\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqq9\" (UniqueName: \"kubernetes.io/projected/242f0489-8ae7-42e5-bb8b-77d84a5fa711-kube-api-access-5fqq9\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-multus-certs\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973223 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6c8a0fa-aca4-466b-a901-7e785f569215-ovn-node-metrics-cert\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-host-slash\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-device-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.973317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-modprobe-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973336 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-cnibin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-netns\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973389 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973415 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-iptables-alerter-script\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973418 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-var-lib-kubelet\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kmx\" (UniqueName: \"kubernetes.io/projected/0856a836-c4e8-4363-bcb1-aaa671091730-kube-api-access-s9kmx\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973310 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-socket-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-var-lib-kubelet\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973522 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-host-slash\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.973576 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973630 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-modprobe-d\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.973652 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:14.473630999 +0000 UTC m=+3.011858648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973583 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996cb3b4-342a-44e6-8e35-60887aab2638-tmp-dir\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.973769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973750 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysctl-conf\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-device-dir\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996cb3b4-342a-44e6-8e35-60887aab2638-tmp-dir\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973865 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973899 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-k8s-cni-cncf-io\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-etc-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-env-overrides\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.973980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-dbus\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-sys-fs\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974089 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-cnibin\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974102 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-sys-fs\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-tmp\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-kubelet-config\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974216 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysconfig\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974223 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-kubelet-config\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:13.974287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-sys\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974267 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-lib-modules\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a67104f9-4e22-41f3-aa18-5f401c8503be-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-system-cni-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-etc-sysconfig\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-system-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-sys\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974436 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a67104f9-4e22-41f3-aa18-5f401c8503be-system-cni-dir\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.975261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.974541 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0856a836-c4e8-4363-bcb1-aaa671091730-lib-modules\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.977009 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.976977 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-tmp\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.977114 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.977036 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0856a836-c4e8-4363-bcb1-aaa671091730-etc-tuned\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.988305 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.988279 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:13.988447 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.988312 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:13.988447 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.988369 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:13.988581 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:13.988565 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:14.488544306 +0000 UTC m=+3.026771952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:13.990060 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.990028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4ms4\" (UniqueName: \"kubernetes.io/projected/beeb3439-9bfd-44d4-b9cb-5f5253c8d607-kube-api-access-g4ms4\") pod \"iptables-alerter-js2hr\" (UID: \"beeb3439-9bfd-44d4-b9cb-5f5253c8d607\") " pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:13.990698 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.990672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqn8w\" (UniqueName: \"kubernetes.io/projected/996cb3b4-342a-44e6-8e35-60887aab2638-kube-api-access-bqn8w\") pod \"node-resolver-7tcvq\" (UID: \"996cb3b4-342a-44e6-8e35-60887aab2638\") " pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:13.991190 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.991168 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kmx\" (UniqueName: \"kubernetes.io/projected/0856a836-c4e8-4363-bcb1-aaa671091730-kube-api-access-s9kmx\") pod \"tuned-tl4cq\" (UID: \"0856a836-c4e8-4363-bcb1-aaa671091730\") " pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:13.991399 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.991381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cj4\" (UniqueName: \"kubernetes.io/projected/a67104f9-4e22-41f3-aa18-5f401c8503be-kube-api-access-t8cj4\") pod \"multus-additional-cni-plugins-lx6kw\" (UID: \"a67104f9-4e22-41f3-aa18-5f401c8503be\") " pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:13.991793 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:13.991777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4mm\" (UniqueName: \"kubernetes.io/projected/91515f2d-1dd7-4033-98a3-ba6fb563bb2d-kube-api-access-wc4mm\") pod \"aws-ebs-csi-driver-node-qb8lq\" (UID: \"91515f2d-1dd7-4033-98a3-ba6fb563bb2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:14.050167 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.050135 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 18:44:14.075123 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075090 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/242f0489-8ae7-42e5-bb8b-77d84a5fa711-host\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-etc-kubernetes\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-systemd-units\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075173 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/242f0489-8ae7-42e5-bb8b-77d84a5fa711-host\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075192 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mg4v\" (UniqueName: \"kubernetes.io/projected/e6c8a0fa-aca4-466b-a901-7e785f569215-kube-api-access-9mg4v\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075230 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-etc-kubernetes\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075233 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqq9\" (UniqueName: \"kubernetes.io/projected/242f0489-8ae7-42e5-bb8b-77d84a5fa711-kube-api-access-5fqq9\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075254 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-systemd-units\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-multus-certs\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6c8a0fa-aca4-466b-a901-7e785f569215-ovn-node-metrics-cert\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-cnibin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-netns\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-cnibin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075408 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-k8s-cni-cncf-io\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075427 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-multus-certs\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-etc-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075447 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-netns\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-run-k8s-cni-cncf-io\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-etc-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075505 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075522 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-env-overrides\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075553 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-system-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-cni-binary-copy\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-system-cni-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.075734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075609 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-multus\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-hostroot\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe239cdd-811b-4875-92d6-21ecf3c41138-konnectivity-ca\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075700 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-hostroot\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52js7\" (UniqueName: \"kubernetes.io/projected/5c44670c-587b-4929-8898-672eb5b2ad8b-kube-api-access-52js7\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075732 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-multus\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075746 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-config\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-os-release\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-kubelet\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-conf-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/82292e4a-5143-4fd1-8750-b8502a4041d6-kube-api-access-m7rbf\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-slash\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-os-release\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-netns\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075941 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-env-overrides\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075973 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-netns\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075978 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-conf-dir\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.076605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.075973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-systemd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076027 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-kubelet\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076034 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-slash\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-node-log\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-node-log\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076082 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-systemd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-script-lib\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.076101 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-daemon-config\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076140 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-bin\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe239cdd-811b-4875-92d6-21ecf3c41138-agent-certs\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-bin\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.076274 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:14.57623076 +0000 UTC m=+3.114458394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-ovn\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076325 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-netd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-config\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-log-socket\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076441 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/242f0489-8ae7-42e5-bb8b-77d84a5fa711-serviceca\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076464 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-socket-dir-parent\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076506 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-kubelet\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-var-lib-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076549 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-run-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-bin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-run-ovn\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076636 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-cni-netd\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076646 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-host-var-lib-cni-bin\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-log-socket\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076694 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076693 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6c8a0fa-aca4-466b-a901-7e785f569215-ovnkube-script-lib\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-daemon-config\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c44670c-587b-4929-8898-672eb5b2ad8b-multus-socket-dir-parent\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-host-kubelet\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.077905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076772 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe239cdd-811b-4875-92d6-21ecf3c41138-konnectivity-ca\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:14.078372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.076786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6c8a0fa-aca4-466b-a901-7e785f569215-var-lib-openvswitch\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.078372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.077050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c44670c-587b-4929-8898-672eb5b2ad8b-cni-binary-copy\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.078372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.077086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/242f0489-8ae7-42e5-bb8b-77d84a5fa711-serviceca\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.078727 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.078704 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6c8a0fa-aca4-466b-a901-7e785f569215-ovn-node-metrics-cert\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.079144 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.079123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe239cdd-811b-4875-92d6-21ecf3c41138-agent-certs\") pod \"konnectivity-agent-whppk\" (UID: \"fe239cdd-811b-4875-92d6-21ecf3c41138\") " pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:14.091732 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.091647 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52js7\" (UniqueName: \"kubernetes.io/projected/5c44670c-587b-4929-8898-672eb5b2ad8b-kube-api-access-52js7\") pod \"multus-gmpwb\" (UID: \"5c44670c-587b-4929-8898-672eb5b2ad8b\") " pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.091951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.091931 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/82292e4a-5143-4fd1-8750-b8502a4041d6-kube-api-access-m7rbf\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:14.092199 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.092181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqq9\" (UniqueName: \"kubernetes.io/projected/242f0489-8ae7-42e5-bb8b-77d84a5fa711-kube-api-access-5fqq9\") pod \"node-ca-zt6dh\" (UID: \"242f0489-8ae7-42e5-bb8b-77d84a5fa711\") " pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.092312 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.092207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mg4v\" (UniqueName: \"kubernetes.io/projected/e6c8a0fa-aca4-466b-a901-7e785f569215-kube-api-access-9mg4v\") pod \"ovnkube-node-7chl4\" (UID: \"e6c8a0fa-aca4-466b-a901-7e785f569215\") " pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.166613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.166572 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tcvq" Jul 31 18:44:14.176676 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.176629 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-js2hr" Jul 31 18:44:14.182508 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.182459 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" Jul 31 18:44:14.190559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.190535 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" Jul 31 18:44:14.198471 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.198442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" Jul 31 18:44:14.207235 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.207205 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:14.215007 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.214983 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zt6dh" Jul 31 18:44:14.221743 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.221724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gmpwb" Jul 31 18:44:14.227577 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.227558 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:14.478903 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.478808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:14.479046 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.478968 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:14.479093 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.479044 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:15.479027181 +0000 UTC m=+4.017254811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:14.579711 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.579668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:14.579907 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.579770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:14.579979 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.579908 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:14.579979 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.579974 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:15.579955715 +0000 UTC m=+4.118183367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:14.580428 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.580403 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:14.580428 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.580429 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:14.580598 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.580442 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:14.580598 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:14.580502 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:15.580473731 +0000 UTC m=+4.118701361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:14.910230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.910195 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:39:13 +0000 UTC" deadline="2028-02-03 22:07:30.476053232 +0000 UTC" Jul 31 18:44:14.910230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:14.910224 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13251h23m15.565832052s" Jul 31 18:44:14.973013 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.972982 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91515f2d_1dd7_4033_98a3_ba6fb563bb2d.slice/crio-561590555ae056ac4fcbf8f5b31bc743443d15ab06af53b69e9fe420b9d23c7c WatchSource:0}: Error finding container 561590555ae056ac4fcbf8f5b31bc743443d15ab06af53b69e9fe420b9d23c7c: Status 404 returned error can't find the container with id 561590555ae056ac4fcbf8f5b31bc743443d15ab06af53b69e9fe420b9d23c7c Jul 31 18:44:14.974063 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.974027 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c8a0fa_aca4_466b_a901_7e785f569215.slice/crio-5d01f59a74fab139d687ede7e3ccbff6a40609927bcaea79f2016c4c27959ad5 WatchSource:0}: Error finding container 5d01f59a74fab139d687ede7e3ccbff6a40609927bcaea79f2016c4c27959ad5: Status 404 returned error can't find the container with id 5d01f59a74fab139d687ede7e3ccbff6a40609927bcaea79f2016c4c27959ad5 Jul 31 18:44:14.975781 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.975655 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242f0489_8ae7_42e5_bb8b_77d84a5fa711.slice/crio-0a8eab781d8c2d13df73b8602bf2964efde6e20f17be68aca626323ce7eae7c4 WatchSource:0}: Error finding container 0a8eab781d8c2d13df73b8602bf2964efde6e20f17be68aca626323ce7eae7c4: Status 404 returned error can't find the container with id 0a8eab781d8c2d13df73b8602bf2964efde6e20f17be68aca626323ce7eae7c4 Jul 31 18:44:14.979406 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.979377 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c44670c_587b_4929_8898_672eb5b2ad8b.slice/crio-c49cec62392f9ad10b4204fb061a1440fbb7a3dba84b5b65e17b00da8e888f16 WatchSource:0}: Error finding container c49cec62392f9ad10b4204fb061a1440fbb7a3dba84b5b65e17b00da8e888f16: Status 404 returned error can't find the container with id c49cec62392f9ad10b4204fb061a1440fbb7a3dba84b5b65e17b00da8e888f16 Jul 31 18:44:14.980586 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.980562 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67104f9_4e22_41f3_aa18_5f401c8503be.slice/crio-f600fafe39eae493adc898d563df7caf8458bda5f42d91a8bf9f57d15b1f8fd8 WatchSource:0}: Error finding container f600fafe39eae493adc898d563df7caf8458bda5f42d91a8bf9f57d15b1f8fd8: Status 404 returned error can't find the container with id f600fafe39eae493adc898d563df7caf8458bda5f42d91a8bf9f57d15b1f8fd8 Jul 31 18:44:14.981221 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.981199 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeb3439_9bfd_44d4_b9cb_5f5253c8d607.slice/crio-c65eb951aeda5e83800b8b4ae92d7f0a48ee0a8c559553dedf7e4c1a16fcc6ac WatchSource:0}: Error finding container c65eb951aeda5e83800b8b4ae92d7f0a48ee0a8c559553dedf7e4c1a16fcc6ac: Status 404 returned error can't find the container with id c65eb951aeda5e83800b8b4ae92d7f0a48ee0a8c559553dedf7e4c1a16fcc6ac Jul 31 18:44:14.982103 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:44:14.982009 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996cb3b4_342a_44e6_8e35_60887aab2638.slice/crio-febc744167596c2c95769a8949fa90a4a3c7f540deb969f1fd636371b9469106 WatchSource:0}: Error finding container febc744167596c2c95769a8949fa90a4a3c7f540deb969f1fd636371b9469106: Status 404 returned error can't find the container with id febc744167596c2c95769a8949fa90a4a3c7f540deb969f1fd636371b9469106 Jul 31 18:44:15.029159 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.029133 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:15.029268 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.029235 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:15.038808 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.038623 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" event={"ID":"0856a836-c4e8-4363-bcb1-aaa671091730","Type":"ContainerStarted","Data":"fb46d1b7a63e3118a6b289ed550f23a4a26ccf6b2b7c19c17386e6b346fa6887"} Jul 31 18:44:15.039958 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.039926 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-whppk" event={"ID":"fe239cdd-811b-4875-92d6-21ecf3c41138","Type":"ContainerStarted","Data":"10f4714a59fc97a6b23ffeed0e6ddbb223b5ecb0247ed53a3dcb95aa8ed35ad3"} Jul 31 18:44:15.041748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.041725 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gmpwb" event={"ID":"5c44670c-587b-4929-8898-672eb5b2ad8b","Type":"ContainerStarted","Data":"c49cec62392f9ad10b4204fb061a1440fbb7a3dba84b5b65e17b00da8e888f16"} Jul 31 18:44:15.043101 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.043080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zt6dh" event={"ID":"242f0489-8ae7-42e5-bb8b-77d84a5fa711","Type":"ContainerStarted","Data":"0a8eab781d8c2d13df73b8602bf2964efde6e20f17be68aca626323ce7eae7c4"} Jul 31 18:44:15.044244 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.044222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tcvq" event={"ID":"996cb3b4-342a-44e6-8e35-60887aab2638","Type":"ContainerStarted","Data":"febc744167596c2c95769a8949fa90a4a3c7f540deb969f1fd636371b9469106"} Jul 31 18:44:15.045590 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.045554 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-js2hr" event={"ID":"beeb3439-9bfd-44d4-b9cb-5f5253c8d607","Type":"ContainerStarted","Data":"c65eb951aeda5e83800b8b4ae92d7f0a48ee0a8c559553dedf7e4c1a16fcc6ac"} Jul 31 18:44:15.046777 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.046749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerStarted","Data":"f600fafe39eae493adc898d563df7caf8458bda5f42d91a8bf9f57d15b1f8fd8"} Jul 31 18:44:15.047912 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.047881 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"5d01f59a74fab139d687ede7e3ccbff6a40609927bcaea79f2016c4c27959ad5"} Jul 31 18:44:15.048965 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.048945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" event={"ID":"91515f2d-1dd7-4033-98a3-ba6fb563bb2d","Type":"ContainerStarted","Data":"561590555ae056ac4fcbf8f5b31bc743443d15ab06af53b69e9fe420b9d23c7c"} Jul 31 18:44:15.486846 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.486165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:15.486846 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.486358 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:15.486846 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.486424 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.486405131 +0000 UTC m=+6.024632765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:15.587248 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.587207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:15.587428 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.587283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:15.587510 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.587458 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:15.587510 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.587477 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:15.587510 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.587505 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:15.587663 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.587565 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.5875455 +0000 UTC m=+6.125773153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:15.587996 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.587971 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:15.588082 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:15.588026 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:17.588010386 +0000 UTC m=+6.126238020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:15.911473 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.911318 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:39:13 +0000 UTC" deadline="2027-12-27 05:28:37.405749157 +0000 UTC" Jul 31 18:44:15.911473 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:15.911378 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12322h44m21.494377052s" Jul 31 18:44:16.031850 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.031808 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:16.032021 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:16.031950 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:16.032343 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.032321 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:16.032434 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:16.032413 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:16.080250 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.080212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" event={"ID":"63165082ff7f5713045b9ebffd02955c","Type":"ContainerStarted","Data":"5c71020255c516335538d672fba93f4fe9d3e0991aec4648938d846d7a887737"} Jul 31 18:44:16.097274 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.097212 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-43.ec2.internal" podStartSLOduration=3.097192967 podStartE2EDuration="3.097192967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:16.097100942 +0000 UTC m=+4.635328596" watchObservedRunningTime="2026-07-31 18:44:16.097192967 +0000 UTC m=+4.635420610" Jul 31 18:44:16.101387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.100640 2566 generic.go:356] "Generic (PLEG): container finished" podID="258b4f1f71dbda6e0fd0f662bb2ecc98" containerID="6f2fe61b98c52a95be0edd1fca4f7f27085f6167f8b595e306fb8965a121f588" exitCode=0 Jul 31 18:44:16.101387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:16.100696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" event={"ID":"258b4f1f71dbda6e0fd0f662bb2ecc98","Type":"ContainerDied","Data":"6f2fe61b98c52a95be0edd1fca4f7f27085f6167f8b595e306fb8965a121f588"} Jul 31 18:44:17.029051 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:17.029018 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:17.029595 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.029146 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:17.121809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:17.121275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" event={"ID":"258b4f1f71dbda6e0fd0f662bb2ecc98","Type":"ContainerStarted","Data":"a534dda8c4ecc7b5f0308d7c06a52ac53a74f93138d6217db99be4cbb94bf598"} Jul 31 18:44:17.504877 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:17.504780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:17.505043 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.504955 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:17.505043 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.505018 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:21.505000347 +0000 UTC m=+10.043227989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:17.606145 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:17.606095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:17.606313 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:17.606177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:17.606425 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606351 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:17.606514 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606433 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:17.606514 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606449 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:17.606514 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606511 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:21.606481508 +0000 UTC m=+10.144709150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:17.606831 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606808 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:17.606900 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:17.606857 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:21.606846472 +0000 UTC m=+10.145074101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:18.031701 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:18.031663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:18.032094 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:18.031716 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:18.032094 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:18.031810 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:18.032094 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:18.031913 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:19.029847 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:19.029370 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:19.029847 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:19.029525 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:20.031842 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:20.030880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:20.031842 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:20.030986 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:20.031842 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:20.031371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:20.031842 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:20.031469 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:21.029518 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:21.029471 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:21.029700 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.029612 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:21.543151 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:21.543113 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:21.543575 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.543297 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:21.543575 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.543376 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:29.543355752 +0000 UTC m=+18.081583386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:21.644017 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:21.643983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:21.644175 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:21.644053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:21.644175 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644149 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:21.644245 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644200 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:21.644245 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644214 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:29.644198579 +0000 UTC m=+18.182426211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:21.644245 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644217 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:21.644245 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644231 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:21.644361 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:21.644280 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:29.644264439 +0000 UTC m=+18.182492082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:22.030177 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:22.030102 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:22.033913 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:22.030652 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:22.033913 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:22.031053 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:22.033913 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:22.031317 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:23.029124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:23.029089 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:23.029634 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:23.029212 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:24.031088 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:24.030608 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:24.031088 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:24.030743 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:24.031809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:24.031666 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:24.031809 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:24.031780 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:25.028993 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:25.028951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:25.029159 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:25.029067 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:26.029141 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:26.029056 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:26.029558 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:26.029197 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:26.029558 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:26.029266 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:26.029558 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:26.029387 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:27.029718 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:27.029683 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:27.030160 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:27.029805 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:28.029301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:28.029262 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:28.029466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:28.029306 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:28.029466 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:28.029391 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:28.029585 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:28.029558 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:29.029643 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:29.029608 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:29.030212 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.029739 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:29.605349 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:29.605315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:29.605520 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.605436 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:29.605520 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.605506 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.605475414 +0000 UTC m=+34.143703043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:29.706630 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:29.706591 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:29.706811 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:29.706682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:29.706811 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706749 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:29.706811 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706769 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:29.706811 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706774 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:29.706811 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706786 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:29.707010 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706831 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.706814227 +0000 UTC m=+34.245041856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:29.707010 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:29.706858 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.70684919 +0000 UTC m=+34.245076819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:30.029894 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:30.029803 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:30.029894 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:30.029821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:30.030369 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:30.029942 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:30.030369 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:30.030075 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:31.029057 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:31.029019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:31.029233 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:31.029142 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:32.030502 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:32.030463 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:32.030928 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:32.030599 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:32.030928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:32.030664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:32.030928 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:32.030799 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:33.029690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:33.029657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:33.029858 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:33.029772 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:34.030017 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.029606 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:34.030638 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.029616 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:34.030638 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:34.030136 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:34.030638 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:34.030228 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:34.156393 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.156356 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tcvq" event={"ID":"996cb3b4-342a-44e6-8e35-60887aab2638","Type":"ContainerStarted","Data":"ce4f9cff05be5ae546d828276dc2768d1dc8c32967b9fb844b0a5852e952a98b"} Jul 31 18:44:34.157870 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.157841 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="08f804a75eb6c419fedebf38efdca99bb9baf55e0d48b4c5d86d72cf36846329" exitCode=0 Jul 31 18:44:34.157958 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.157912 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"08f804a75eb6c419fedebf38efdca99bb9baf55e0d48b4c5d86d72cf36846329"} Jul 31 18:44:34.160800 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.160777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"f296624a2db2746caf0091be0927609a2d20a366bbc2d112ac429b5797698a44"} Jul 31 18:44:34.160891 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.160839 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"8a7baa609fa3c76f4a359e14f7a6f3da4cb5efc56ce5eb07b019f12f61a245b3"} Jul 31 18:44:34.160891 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.160885 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"989c4bda8d9ab44a48f69da94bdaf169c6a217b4ddba09d9441c8d068be21512"} Jul 31 18:44:34.160991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.160928 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"e01093f571a70c1b59fe2978cc60d9c7e0bd1ea95fb3d7b3212f0c5def45067c"} Jul 31 18:44:34.160991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.160963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"859195d089756e66730a7a19f79f27dae21dd34abf61231eaaab24c493b80231"} Jul 31 18:44:34.162397 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.162327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" event={"ID":"91515f2d-1dd7-4033-98a3-ba6fb563bb2d","Type":"ContainerStarted","Data":"d60faadbdcf416a35c439b4d9b6cae4d587cd39d2059835356ee427b0ed4207f"} Jul 31 18:44:34.163835 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.163807 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" event={"ID":"0856a836-c4e8-4363-bcb1-aaa671091730","Type":"ContainerStarted","Data":"ec4817e7796b68466d304facc5ae4c6ce602c5a07f877f00e645b9a19bae212e"} Jul 31 18:44:34.165241 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.165221 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-whppk" event={"ID":"fe239cdd-811b-4875-92d6-21ecf3c41138","Type":"ContainerStarted","Data":"9dba8b79b47dea528681ac7f7a6a099e5e6af82ccd5a04291815a9e383e1becd"} Jul 31 18:44:34.166827 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.166805 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gmpwb" event={"ID":"5c44670c-587b-4929-8898-672eb5b2ad8b","Type":"ContainerStarted","Data":"8f8ce81daabc0fc364a126e970b38216db7dd62372fc81ac9c20ee03c1bb978b"} Jul 31 18:44:34.168125 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.168103 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zt6dh" event={"ID":"242f0489-8ae7-42e5-bb8b-77d84a5fa711","Type":"ContainerStarted","Data":"8b14272c5a2aeeb8a9a08258022264490a474e8aab9e9e62f7fe7d6da34fee3a"} Jul 31 18:44:34.184378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.184332 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-43.ec2.internal" podStartSLOduration=21.184320381 podStartE2EDuration="21.184320381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:17.141153454 +0000 UTC m=+5.679381106" watchObservedRunningTime="2026-07-31 18:44:34.184320381 +0000 UTC m=+22.722548031" Jul 31 18:44:34.223122 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.223072 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7tcvq" podStartSLOduration=3.6561889340000002 podStartE2EDuration="22.223056874s" totalImagesPullingTime="18.56686794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:34.184851581 +0000 UTC m=+22.723079232" watchObservedRunningTime="2026-07-31 18:44:34.223056874 +0000 UTC m=+22.761284525" Jul 31 18:44:34.257365 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.257317 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tl4cq" podStartSLOduration=3.692866565 podStartE2EDuration="22.257301083s" totalImagesPullingTime="18.564434518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:34.223194451 +0000 UTC m=+22.761422088" watchObservedRunningTime="2026-07-31 18:44:34.257301083 +0000 UTC m=+22.795528734" Jul 31 18:44:34.257821 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.257793 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-whppk" podStartSLOduration=11.468745687 podStartE2EDuration="22.257785582s" totalImagesPullingTime="10.789039895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:34.257436718 +0000 UTC m=+22.795664351" watchObservedRunningTime="2026-07-31 18:44:34.257785582 +0000 UTC m=+22.796013232" Jul 31 18:44:34.281782 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.281740 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zt6dh" podStartSLOduration=3.7088519829999997 podStartE2EDuration="22.281725259s" totalImagesPullingTime="18.572873276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:34.281339869 +0000 UTC m=+22.819567522" watchObservedRunningTime="2026-07-31 18:44:34.281725259 +0000 UTC m=+22.819952909" Jul 31 18:44:34.331651 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:34.331591 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gmpwb" podStartSLOduration=3.700017408 podStartE2EDuration="22.331570447s" totalImagesPullingTime="18.631553039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:34.331091247 +0000 UTC m=+22.869318899" watchObservedRunningTime="2026-07-31 18:44:34.331570447 +0000 UTC m=+22.869798116" Jul 31 18:44:35.029407 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.029376 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:35.029717 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:35.029500 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:35.173162 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.173130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"065f7809880dd9ee2cd2d7419ed22a3c604dc9cdefbfa3264b28f82556f5eae9"} Jul 31 18:44:35.174331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.174312 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-js2hr" event={"ID":"beeb3439-9bfd-44d4-b9cb-5f5253c8d607","Type":"ContainerStarted","Data":"9d8d8bab9a12554b440fa7a3565ea880dcefdeaf779344439d33e4fcee50dca9"} Jul 31 18:44:35.213054 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.213000 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-js2hr" podStartSLOduration=4.644736507 podStartE2EDuration="23.212980269s" totalImagesPullingTime="18.568243762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:35.211955701 +0000 UTC m=+23.750183351" watchObservedRunningTime="2026-07-31 18:44:35.212980269 +0000 UTC m=+23.751207921" Jul 31 18:44:35.313290 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.313262 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 18:44:35.945552 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.945432 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T18:44:35.313283791Z","UUID":"3fe9a3df-cb25-4a07-8595-72464549b653","Handler":null,"Name":"","Endpoint":""} Jul 31 18:44:35.947124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.947095 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 18:44:35.947237 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:35.947138 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 18:44:36.029809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.029779 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:36.029989 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:36.029925 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:36.030069 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.029988 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:36.030125 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:36.030098 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:36.033062 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.033033 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:36.033184 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.033103 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:36.178826 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.178782 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" event={"ID":"91515f2d-1dd7-4033-98a3-ba6fb563bb2d","Type":"ContainerStarted","Data":"78cf13ede6610fc05bc35f15cffbb42bed91d3b697c51d302adef98c1d703d72"} Jul 31 18:44:36.179293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.179057 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:36.179483 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:36.179467 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-whppk" Jul 31 18:44:37.029698 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:37.029460 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:37.029871 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:37.029754 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:37.185286 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:37.185242 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"ae3fde09456aa79be82770a8b460390abb473e3d064b0aff62587031b0bd9f87"} Jul 31 18:44:37.187748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:37.187691 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" event={"ID":"91515f2d-1dd7-4033-98a3-ba6fb563bb2d","Type":"ContainerStarted","Data":"121225ebb7a749e1b7809e38891d4a16a7f7fd854e63d898fe6775203d15979c"} Jul 31 18:44:37.211589 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:37.211545 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qb8lq" podStartSLOduration=11.562676246 podStartE2EDuration="25.211532767s" totalImagesPullingTime="13.648856521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:37.211403444 +0000 UTC m=+25.749631095" watchObservedRunningTime="2026-07-31 18:44:37.211532767 +0000 UTC m=+25.749760417" Jul 31 18:44:38.032870 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:38.032841 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:38.033079 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:38.032961 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:38.033079 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:38.033012 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:38.033208 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:38.033091 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:39.029099 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:39.029033 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:39.029664 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:39.029133 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:39.375372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:39.375347 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tcvq_996cb3b4-342a-44e6-8e35-60887aab2638/dns-node-resolver/0.log" Jul 31 18:44:40.032629 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.032406 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:40.033093 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.032481 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:40.033093 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:40.032650 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:40.033093 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:40.032742 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:40.196700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.196666 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="86fb51a13751ef7d9570f448e32b007a3030951f0ae8c3819ed5ae1c7c1a112c" exitCode=0 Jul 31 18:44:40.196700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.196696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"86fb51a13751ef7d9570f448e32b007a3030951f0ae8c3819ed5ae1c7c1a112c"} Jul 31 18:44:40.200314 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.200292 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" event={"ID":"e6c8a0fa-aca4-466b-a901-7e785f569215","Type":"ContainerStarted","Data":"2738a7ffb24ccdcbe89151c5d34b783fa38a2bcd058ce503708ee7906ee94710"} Jul 31 18:44:40.200613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.200592 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:40.200685 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.200648 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:40.216357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.216338 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:40.264896 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.264847 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" podStartSLOduration=9.631679578 podStartE2EDuration="28.264832265s" totalImagesPullingTime="18.633152687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:40.264297499 +0000 UTC m=+28.802525141" watchObservedRunningTime="2026-07-31 18:44:40.264832265 +0000 UTC m=+28.803059917" Jul 31 18:44:40.755225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:40.755197 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zt6dh_242f0489-8ae7-42e5-bb8b-77d84a5fa711/node-ca/0.log" Jul 31 18:44:41.029329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.029260 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:41.029461 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:41.029362 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:41.203184 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.203152 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:41.218176 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.218148 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:44:41.739698 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.739537 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nswnl"] Jul 31 18:44:41.739879 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.739814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:41.739943 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:41.739916 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:41.740840 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.740814 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rfnxc"] Jul 31 18:44:41.740961 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.740901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:41.741014 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:41.740994 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:41.741426 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.741403 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6gr7q"] Jul 31 18:44:41.741522 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:41.741507 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:41.741592 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:41.741574 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:42.209825 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:42.209794 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="a46142176b9a37a521348a15e2cfeb850c770f2c017f773a1957a9ca0c21bb9d" exitCode=0 Jul 31 18:44:42.210208 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:42.209872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"a46142176b9a37a521348a15e2cfeb850c770f2c017f773a1957a9ca0c21bb9d"} Jul 31 18:44:43.028974 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:43.028953 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:43.029064 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:43.029047 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:43.029107 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:43.029075 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:43.029143 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:43.029124 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:43.214902 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:43.214867 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="f0832bb25e1429f4e323f63a637dd60b2c23b6a4ceef00f1979fa414cd9c6a93" exitCode=0 Jul 31 18:44:43.215376 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:43.214929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"f0832bb25e1429f4e323f63a637dd60b2c23b6a4ceef00f1979fa414cd9c6a93"} Jul 31 18:44:44.032990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:44.032960 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:44.033123 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:44.033080 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:45.029843 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:45.029805 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:45.030461 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.029938 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:45.030461 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:45.029999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:45.030461 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.030115 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:45.636750 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:45.636714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:45.636970 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.636870 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:45.636970 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.636932 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret podName:b178f3a3-fa58-4d04-9e33-bf0533dc0aea nodeName:}" failed. No retries permitted until 2026-07-31 18:45:17.636917429 +0000 UTC m=+66.175145058 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret") pod "global-pull-secret-syncer-6gr7q" (UID: "b178f3a3-fa58-4d04-9e33-bf0533dc0aea") : object "kube-system"/"original-pull-secret" not registered Jul 31 18:44:45.737167 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:45.737137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:45.737345 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:45.737182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:45.737345 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737301 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:45.737345 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737317 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 18:44:45.737345 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737337 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 18:44:45.737549 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737349 2566 projected.go:196] Error preparing data for projected volume kube-api-access-9n9hm for pod openshift-network-diagnostics/network-check-target-rfnxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:45.737549 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737373 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs podName:82292e4a-5143-4fd1-8750-b8502a4041d6 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:17.737353422 +0000 UTC m=+66.275581054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs") pod "network-metrics-daemon-nswnl" (UID: "82292e4a-5143-4fd1-8750-b8502a4041d6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 18:44:45.737549 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:45.737397 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm podName:a741de37-d1d8-4f52-b018-e8d38787f906 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:17.737387136 +0000 UTC m=+66.275614807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9n9hm" (UniqueName: "kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm") pod "network-check-target-rfnxc" (UID: "a741de37-d1d8-4f52-b018-e8d38787f906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 18:44:46.029139 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:46.029048 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:46.029293 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:46.029171 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:47.029451 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:47.029403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:47.029451 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:47.029436 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:47.030061 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:47.029543 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:47.030061 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:47.029658 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:48.029757 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:48.029721 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:48.030146 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:48.029848 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:48.229663 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:48.229627 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="6fa4383f6b321a4424976b09d902cf16a3ee9cfae476cfb1ab0535aa31ddec5b" exitCode=0 Jul 31 18:44:48.229833 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:48.229690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"6fa4383f6b321a4424976b09d902cf16a3ee9cfae476cfb1ab0535aa31ddec5b"} Jul 31 18:44:49.028841 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:49.028811 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:49.029009 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:49.028812 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:49.029009 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:49.028913 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:49.029009 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:49.028988 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:49.234663 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:49.234627 2566 generic.go:356] "Generic (PLEG): container finished" podID="a67104f9-4e22-41f3-aa18-5f401c8503be" containerID="3228c382ba955fb701e8beb5405628d1441729390aeb0af27de21e63a7e23d77" exitCode=0 Jul 31 18:44:49.235031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:49.234687 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerDied","Data":"3228c382ba955fb701e8beb5405628d1441729390aeb0af27de21e63a7e23d77"} Jul 31 18:44:50.029723 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:50.029540 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:50.029904 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:50.029820 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:50.240216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:50.240186 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" event={"ID":"a67104f9-4e22-41f3-aa18-5f401c8503be","Type":"ContainerStarted","Data":"52957221fac2a6d2e221cf77f1012fdbda6c8dfc9874d94c84626f94947361a2"} Jul 31 18:44:50.268409 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:50.268359 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lx6kw" podStartSLOduration=8.212357955 podStartE2EDuration="38.26834101s" totalImagesPullingTime="30.055983055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:50.268077862 +0000 UTC m=+38.806305513" watchObservedRunningTime="2026-07-31 18:44:50.26834101 +0000 UTC m=+38.806568666" Jul 31 18:44:51.028944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:51.028907 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:51.029158 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:51.029032 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:51.029158 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:51.029072 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:51.029158 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:51.029145 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:52.029944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:52.029908 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:52.030310 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:52.030007 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:53.029115 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:53.029080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:53.029115 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:53.029118 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:53.029332 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:53.029192 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:53.029332 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:53.029304 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:54.029590 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:54.029560 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:54.029924 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:54.029672 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:55.029187 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:55.029154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:55.029381 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:55.029260 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:55.029381 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:55.029311 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:55.029471 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:55.029391 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:56.028881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:56.028845 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:56.029335 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:56.028991 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:57.028933 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:57.028896 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:57.029298 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:57.028896 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:57.029298 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:57.029000 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:44:57.029298 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:57.029078 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:58.029224 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:58.029187 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:44:58.029611 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:58.029335 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:44:59.029067 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:59.029036 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:44:59.029216 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:59.029130 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:44:59.029216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:44:59.029030 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:44:59.029216 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:44:59.029204 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:45:00.029735 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:00.029693 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:00.030109 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:00.029833 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:45:01.029057 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:01.029017 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:01.029315 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:01.029017 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:45:01.029315 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:01.029130 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rfnxc" podUID="a741de37-d1d8-4f52-b018-e8d38787f906" Jul 31 18:45:01.029315 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:01.029203 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6gr7q" podUID="b178f3a3-fa58-4d04-9e33-bf0533dc0aea" Jul 31 18:45:02.029631 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.029596 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:02.030026 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:02.029688 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nswnl" podUID="82292e4a-5143-4fd1-8750-b8502a4041d6" Jul 31 18:45:02.257783 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.257757 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-43.ec2.internal" event="NodeReady" Jul 31 18:45:02.257944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.257875 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 18:45:02.308434 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.308354 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5jd6f"] Jul 31 18:45:02.322579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.322541 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2bh6g"] Jul 31 18:45:02.322755 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.322656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.325545 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.325520 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 18:45:02.325670 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.325520 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 18:45:02.325670 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.325521 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8wjd\"" Jul 31 18:45:02.337866 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.337843 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jd6f"] Jul 31 18:45:02.337997 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.337879 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bh6g"] Jul 31 18:45:02.337997 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.337965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.340010 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.339988 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5l2mc"] Jul 31 18:45:02.340815 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.340797 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 18:45:02.340901 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.340797 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96c4f\"" Jul 31 18:45:02.340964 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.340799 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 18:45:02.340964 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.340798 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 18:45:02.355745 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.355720 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5l2mc"] Jul 31 18:45:02.355903 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.355828 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.357310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357290 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9a045f5-fe8e-4839-b71c-f9c671da2cae-metrics-tls\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.357456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a045f5-fe8e-4839-b71c-f9c671da2cae-tmp-dir\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.357456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcx9\" (UniqueName: \"kubernetes.io/projected/c9a045f5-fe8e-4839-b71c-f9c671da2cae-kube-api-access-rhcx9\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.357456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrdf\" (UniqueName: \"kubernetes.io/projected/a5014d93-4ec1-4abd-a819-cdc21e81722e-kube-api-access-jsrdf\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.357456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a045f5-fe8e-4839-b71c-f9c671da2cae-config-volume\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.357691 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.357480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5014d93-4ec1-4abd-a819-cdc21e81722e-cert\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.358708 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.358678 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7tf4\"" Jul 31 18:45:02.358809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.358708 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 18:45:02.358809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.358787 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 18:45:02.358907 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.358821 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 18:45:02.359248 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.359236 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 18:45:02.458043 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458000 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9a045f5-fe8e-4839-b71c-f9c671da2cae-metrics-tls\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-crio-socket\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a045f5-fe8e-4839-b71c-f9c671da2cae-tmp-dir\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcx9\" (UniqueName: \"kubernetes.io/projected/c9a045f5-fe8e-4839-b71c-f9c671da2cae-kube-api-access-rhcx9\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.458216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcft8\" (UniqueName: \"kubernetes.io/projected/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-api-access-bcft8\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.458464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrdf\" (UniqueName: \"kubernetes.io/projected/a5014d93-4ec1-4abd-a819-cdc21e81722e-kube-api-access-jsrdf\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.458464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458320 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a045f5-fe8e-4839-b71c-f9c671da2cae-config-volume\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.458464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-data-volume\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.458464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5014d93-4ec1-4abd-a819-cdc21e81722e-cert\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.458976 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9a045f5-fe8e-4839-b71c-f9c671da2cae-config-volume\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.459112 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.458994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a045f5-fe8e-4839-b71c-f9c671da2cae-tmp-dir\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.462440 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.462417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9a045f5-fe8e-4839-b71c-f9c671da2cae-metrics-tls\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.462675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.462461 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5014d93-4ec1-4abd-a819-cdc21e81722e-cert\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.467836 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.467799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcx9\" (UniqueName: \"kubernetes.io/projected/c9a045f5-fe8e-4839-b71c-f9c671da2cae-kube-api-access-rhcx9\") pod \"dns-default-5jd6f\" (UID: \"c9a045f5-fe8e-4839-b71c-f9c671da2cae\") " pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.468049 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.468028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrdf\" (UniqueName: \"kubernetes.io/projected/a5014d93-4ec1-4abd-a819-cdc21e81722e-kube-api-access-jsrdf\") pod \"ingress-canary-2bh6g\" (UID: \"a5014d93-4ec1-4abd-a819-cdc21e81722e\") " pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.559587 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559473 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcft8\" (UniqueName: \"kubernetes.io/projected/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-api-access-bcft8\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.559587 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-data-volume\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.559587 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.559856 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559620 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.559856 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-crio-socket\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.559856 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559790 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-crio-socket\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.560009 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.559986 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-data-volume\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.560171 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.560156 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.562084 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.562066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.568382 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.568358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcft8\" (UniqueName: \"kubernetes.io/projected/4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970-kube-api-access-bcft8\") pod \"insights-runtime-extractor-5l2mc\" (UID: \"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970\") " pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.631987 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.631950 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:02.646053 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.646024 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bh6g" Jul 31 18:45:02.665177 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.665158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5l2mc" Jul 31 18:45:02.865564 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.865537 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jd6f"] Jul 31 18:45:02.870105 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:02.870060 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a045f5_fe8e_4839_b71c_f9c671da2cae.slice/crio-2d2777458f8adbabcc2824e9b46a9708425ed7c445ddf6409a86c41d0dc27c8e WatchSource:0}: Error finding container 2d2777458f8adbabcc2824e9b46a9708425ed7c445ddf6409a86c41d0dc27c8e: Status 404 returned error can't find the container with id 2d2777458f8adbabcc2824e9b46a9708425ed7c445ddf6409a86c41d0dc27c8e Jul 31 18:45:02.880853 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:02.880829 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5l2mc"] Jul 31 18:45:02.884177 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:02.884151 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a1a5a_6752_4c8e_ab3b_8e0a5868b970.slice/crio-8e8fbeada449c89509b938242d319bb780bfa0bfebc1281c3fe3c850051b3965 WatchSource:0}: Error finding container 8e8fbeada449c89509b938242d319bb780bfa0bfebc1281c3fe3c850051b3965: Status 404 returned error can't find the container with id 8e8fbeada449c89509b938242d319bb780bfa0bfebc1281c3fe3c850051b3965 Jul 31 18:45:03.029858 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.029820 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:03.030267 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.030010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:45:03.033307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.033284 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 18:45:03.033432 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.033360 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\"" Jul 31 18:45:03.033611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.033596 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 18:45:03.033611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.033609 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 18:45:03.086037 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.085935 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bh6g"] Jul 31 18:45:03.090829 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:03.090800 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5014d93_4ec1_4abd_a819_cdc21e81722e.slice/crio-ae16d1088ab614c99a641bad0e1c195eabea693c5d997a4b03ba73b976dbcd68 WatchSource:0}: Error finding container ae16d1088ab614c99a641bad0e1c195eabea693c5d997a4b03ba73b976dbcd68: Status 404 returned error can't find the container with id ae16d1088ab614c99a641bad0e1c195eabea693c5d997a4b03ba73b976dbcd68 Jul 31 18:45:03.269872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.269833 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5l2mc" event={"ID":"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970","Type":"ContainerStarted","Data":"e18e33e9e414991a68ef5fd62bf9c1e0019b20ebdc0cd7ed1ac0e9d06a70f20c"} Jul 31 18:45:03.270029 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.269900 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5l2mc" event={"ID":"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970","Type":"ContainerStarted","Data":"8e8fbeada449c89509b938242d319bb780bfa0bfebc1281c3fe3c850051b3965"} Jul 31 18:45:03.270977 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.270951 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bh6g" event={"ID":"a5014d93-4ec1-4abd-a819-cdc21e81722e","Type":"ContainerStarted","Data":"ae16d1088ab614c99a641bad0e1c195eabea693c5d997a4b03ba73b976dbcd68"} Jul 31 18:45:03.271954 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:03.271933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jd6f" event={"ID":"c9a045f5-fe8e-4839-b71c-f9c671da2cae","Type":"ContainerStarted","Data":"2d2777458f8adbabcc2824e9b46a9708425ed7c445ddf6409a86c41d0dc27c8e"} Jul 31 18:45:04.030943 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:04.029709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:04.034980 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:04.034953 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\"" Jul 31 18:45:04.035206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:04.035033 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 18:45:06.281710 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.281675 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5l2mc" event={"ID":"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970","Type":"ContainerStarted","Data":"c6102c1408d431d0e66eab630d42e2f78500b0daa8943b9b4997b24d4c1829e7"} Jul 31 18:45:06.283178 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.283159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bh6g" event={"ID":"a5014d93-4ec1-4abd-a819-cdc21e81722e","Type":"ContainerStarted","Data":"f750eca68c9a0d5695971b4757f3e67582a93e0086329a90a50d9a743bb3bf8f"} Jul 31 18:45:06.284781 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.284760 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jd6f" event={"ID":"c9a045f5-fe8e-4839-b71c-f9c671da2cae","Type":"ContainerStarted","Data":"12193845b8a20cf062f637d4baeb7ae124c9f370b82eed9aa88d018bdfb63c1d"} Jul 31 18:45:06.284859 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.284801 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jd6f" event={"ID":"c9a045f5-fe8e-4839-b71c-f9c671da2cae","Type":"ContainerStarted","Data":"31202b85aa20218f3fb8a622223e7bbb3cebab3af9325ef94d19d364a942b79e"} Jul 31 18:45:06.284901 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.284885 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:06.306123 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.306074 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2bh6g" podStartSLOduration=1.609682534 podStartE2EDuration="4.306055557s" totalImagesPullingTime="2.696373023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:06.30447243 +0000 UTC m=+54.842700077" watchObservedRunningTime="2026-07-31 18:45:06.306055557 +0000 UTC m=+54.844283187" Jul 31 18:45:06.333939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:06.333881 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5jd6f" podStartSLOduration=1.422903555 podStartE2EDuration="4.333864751s" totalImagesPullingTime="2.910961196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:06.332574844 +0000 UTC m=+54.870802486" watchObservedRunningTime="2026-07-31 18:45:06.333864751 +0000 UTC m=+54.872092402" Jul 31 18:45:08.291675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:08.291635 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5l2mc" event={"ID":"4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970","Type":"ContainerStarted","Data":"9bc7d6a813f84f40044c856bf9a8e97ea56ac920552d42c55f2b9fd88a5a1aa2"} Jul 31 18:45:08.314106 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:08.314056 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5l2mc" podStartSLOduration=1.890930145 podStartE2EDuration="6.314037401s" totalImagesPullingTime="4.423107256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:45:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:08.313035586 +0000 UTC m=+56.851263238" watchObservedRunningTime="2026-07-31 18:45:08.314037401 +0000 UTC m=+56.852265056" Jul 31 18:45:10.696381 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.696345 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg"] Jul 31 18:45:10.722161 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.722132 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg"] Jul 31 18:45:10.722329 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.722247 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:10.726521 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.726469 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 31 18:45:10.726642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.726475 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h54gm\"" Jul 31 18:45:10.827510 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.827444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3fe4c28-933d-4370-a3d3-6271a4764176-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-qf2hg\" (UID: \"f3fe4c28-933d-4370-a3d3-6271a4764176\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:10.928453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.928410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3fe4c28-933d-4370-a3d3-6271a4764176-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-qf2hg\" (UID: \"f3fe4c28-933d-4370-a3d3-6271a4764176\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:10.930917 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:10.930896 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f3fe4c28-933d-4370-a3d3-6271a4764176-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-qf2hg\" (UID: \"f3fe4c28-933d-4370-a3d3-6271a4764176\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:11.030520 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:11.030411 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:11.178678 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:11.178648 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg"] Jul 31 18:45:11.182085 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:11.182050 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fe4c28_933d_4370_a3d3_6271a4764176.slice/crio-eb8b773f96f6df9ea2a615a647601517cd9017cf6843f2ddef14eedfeddbc39a WatchSource:0}: Error finding container eb8b773f96f6df9ea2a615a647601517cd9017cf6843f2ddef14eedfeddbc39a: Status 404 returned error can't find the container with id eb8b773f96f6df9ea2a615a647601517cd9017cf6843f2ddef14eedfeddbc39a Jul 31 18:45:11.303145 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:11.303067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" event={"ID":"f3fe4c28-933d-4370-a3d3-6271a4764176","Type":"ContainerStarted","Data":"eb8b773f96f6df9ea2a615a647601517cd9017cf6843f2ddef14eedfeddbc39a"} Jul 31 18:45:13.228845 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.228814 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7chl4" Jul 31 18:45:13.310605 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.310563 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" event={"ID":"f3fe4c28-933d-4370-a3d3-6271a4764176","Type":"ContainerStarted","Data":"cf9d6837700f467cd4aee6f196c40ebfc6cd2d881abe2d8cc70d673dd57e0a49"} Jul 31 18:45:13.310770 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.310691 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:13.315378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.315353 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" Jul 31 18:45:13.342852 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.342794 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-qf2hg" podStartSLOduration=2.210652993 podStartE2EDuration="3.342778766s" totalImagesPullingTime="1.132125773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:13.342239852 +0000 UTC m=+61.880467503" watchObservedRunningTime="2026-07-31 18:45:13.342778766 +0000 UTC m=+61.881006417" Jul 31 18:45:13.754982 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.754949 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gg8zs"] Jul 31 18:45:13.756924 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.756910 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.761437 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.761413 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 18:45:13.761576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.761459 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 18:45:13.761652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.761610 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 18:45:13.761910 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.761880 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 31 18:45:13.761910 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.761902 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 31 18:45:13.762069 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.762003 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pqhdf\"" Jul 31 18:45:13.770546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.770526 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gg8zs"] Jul 31 18:45:13.850202 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.850164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.850367 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.850213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3ac65c-394a-4b33-ae59-6148d7853d46-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.850367 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.850264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59p4n\" (UniqueName: \"kubernetes.io/projected/7f3ac65c-394a-4b33-ae59-6148d7853d46-kube-api-access-59p4n\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.850367 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.850325 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.951111 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.951071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59p4n\" (UniqueName: \"kubernetes.io/projected/7f3ac65c-394a-4b33-ae59-6148d7853d46-kube-api-access-59p4n\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.951261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.951140 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.951261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.951169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.951261 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.951187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3ac65c-394a-4b33-ae59-6148d7853d46-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.951904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.951873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3ac65c-394a-4b33-ae59-6148d7853d46-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.953597 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.953579 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.953667 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.953648 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f3ac65c-394a-4b33-ae59-6148d7853d46-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:13.961865 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:13.961838 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59p4n\" (UniqueName: \"kubernetes.io/projected/7f3ac65c-394a-4b33-ae59-6148d7853d46-kube-api-access-59p4n\") pod \"prometheus-operator-758d7cb874-gg8zs\" (UID: \"7f3ac65c-394a-4b33-ae59-6148d7853d46\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:14.066368 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.066281 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" Jul 31 18:45:14.197507 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.197460 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-gg8zs"] Jul 31 18:45:14.201116 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:14.201092 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3ac65c_394a_4b33_ae59_6148d7853d46.slice/crio-8c6d810aab78e47ec0bcc790266b702b63795695eaf1832cac61d3bee5c2e4f1 WatchSource:0}: Error finding container 8c6d810aab78e47ec0bcc790266b702b63795695eaf1832cac61d3bee5c2e4f1: Status 404 returned error can't find the container with id 8c6d810aab78e47ec0bcc790266b702b63795695eaf1832cac61d3bee5c2e4f1 Jul 31 18:45:14.205729 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.205712 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:14.208126 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.208113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.211294 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.211278 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 18:45:14.211375 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.211328 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 18:45:14.211375 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.211333 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 18:45:14.212897 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.212882 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 18:45:14.212987 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.212967 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 18:45:14.213141 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.213125 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzdcj\"" Jul 31 18:45:14.213206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.213165 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 18:45:14.213254 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.213209 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 18:45:14.226360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.226335 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:14.253282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.253282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253279 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4zd\" (UniqueName: \"kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.253725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253310 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.253725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253380 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.253725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.253725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.253471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.314925 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.314888 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" event={"ID":"7f3ac65c-394a-4b33-ae59-6148d7853d46","Type":"ContainerStarted","Data":"8c6d810aab78e47ec0bcc790266b702b63795695eaf1832cac61d3bee5c2e4f1"} Jul 31 18:45:14.354894 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.354816 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.354894 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.354858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355062 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.354910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355062 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.354934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4zd\" (UniqueName: \"kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355062 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.354971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.355093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355687 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.355664 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355687 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.355673 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.355813 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.355734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.357598 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.357574 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.357672 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.357638 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.365080 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.365059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4zd\" (UniqueName: \"kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd\") pod \"console-57c987dcf5-7t58c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.518020 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.517979 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:14.647834 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:14.647804 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:14.651084 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:14.651060 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793c842e_a5bc_4ed9_995d_1a8c9c325b2c.slice/crio-ec70555cd74da7403f2f83858843bcc1edee2e3fec69317b7ec6e13d6bf30b1d WatchSource:0}: Error finding container ec70555cd74da7403f2f83858843bcc1edee2e3fec69317b7ec6e13d6bf30b1d: Status 404 returned error can't find the container with id ec70555cd74da7403f2f83858843bcc1edee2e3fec69317b7ec6e13d6bf30b1d Jul 31 18:45:15.326372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:15.326273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c987dcf5-7t58c" event={"ID":"793c842e-a5bc-4ed9-995d-1a8c9c325b2c","Type":"ContainerStarted","Data":"ec70555cd74da7403f2f83858843bcc1edee2e3fec69317b7ec6e13d6bf30b1d"} Jul 31 18:45:16.289891 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:16.289858 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jd6f" Jul 31 18:45:16.335029 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:16.334984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" event={"ID":"7f3ac65c-394a-4b33-ae59-6148d7853d46","Type":"ContainerStarted","Data":"19cb73a8e37add3976c641a0a44278b8b91b10857d70ce3ddad3729ffd0c9f1c"} Jul 31 18:45:16.335467 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:16.335044 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" event={"ID":"7f3ac65c-394a-4b33-ae59-6148d7853d46","Type":"ContainerStarted","Data":"cd096f4a3975f13b02e2765dd090021e2fd3a425c0a0bb491000ae80adb1ec38"} Jul 31 18:45:16.354601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:16.354549 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-gg8zs" podStartSLOduration=1.8881196359999999 podStartE2EDuration="3.354531438s" totalImagesPullingTime="1.466411802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:16.353612609 +0000 UTC m=+64.891840264" watchObservedRunningTime="2026-07-31 18:45:16.354531438 +0000 UTC m=+64.892759088" Jul 31 18:45:17.684548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.684513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:45:17.688175 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.688154 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 18:45:17.697647 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.697628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b178f3a3-fa58-4d04-9e33-bf0533dc0aea-original-pull-secret\") pod \"global-pull-secret-syncer-6gr7q\" (UID: \"b178f3a3-fa58-4d04-9e33-bf0533dc0aea\") " pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:45:17.744546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.744515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6gr7q" Jul 31 18:45:17.785565 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.785535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:17.785703 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.785600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:17.788651 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.788632 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 18:45:17.789934 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.789918 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 18:45:17.798138 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.798112 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82292e4a-5143-4fd1-8750-b8502a4041d6-metrics-certs\") pod \"network-metrics-daemon-nswnl\" (UID: \"82292e4a-5143-4fd1-8750-b8502a4041d6\") " pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:17.798842 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.798828 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 18:45:17.808263 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.808241 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9hm\" (UniqueName: \"kubernetes.io/projected/a741de37-d1d8-4f52-b018-e8d38787f906-kube-api-access-9n9hm\") pod \"network-check-target-rfnxc\" (UID: \"a741de37-d1d8-4f52-b018-e8d38787f906\") " pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:17.846425 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.846395 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\"" Jul 31 18:45:17.854047 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:17.854024 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nswnl" Jul 31 18:45:18.044033 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.044008 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\"" Jul 31 18:45:18.045099 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.045079 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nswnl"] Jul 31 18:45:18.048211 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.048190 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82292e4a_5143_4fd1_8750_b8502a4041d6.slice/crio-f2e7de66cfd445bef3196f8b9c1f53b734147a2ae5118b0e64d4ecafafef951d WatchSource:0}: Error finding container f2e7de66cfd445bef3196f8b9c1f53b734147a2ae5118b0e64d4ecafafef951d: Status 404 returned error can't find the container with id f2e7de66cfd445bef3196f8b9c1f53b734147a2ae5118b0e64d4ecafafef951d Jul 31 18:45:18.049843 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.049824 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:18.059765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.059740 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6gr7q"] Jul 31 18:45:18.063605 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.063583 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb178f3a3_fa58_4d04_9e33_bf0533dc0aea.slice/crio-193b37ad184e3236a3e3fcc32d2754e92560cc25cb9ee6c8119ef2552e9d86f1 WatchSource:0}: Error finding container 193b37ad184e3236a3e3fcc32d2754e92560cc25cb9ee6c8119ef2552e9d86f1: Status 404 returned error can't find the container with id 193b37ad184e3236a3e3fcc32d2754e92560cc25cb9ee6c8119ef2552e9d86f1 Jul 31 18:45:18.136652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.136620 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf"] Jul 31 18:45:18.153392 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.153366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.158507 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.158091 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 31 18:45:18.158507 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.158423 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 31 18:45:18.158684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.158636 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z5sbc\"" Jul 31 18:45:18.158738 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.158685 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf"] Jul 31 18:45:18.171374 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.171345 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ccppf"] Jul 31 18:45:18.177382 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.177358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.182737 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.182710 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbzjz\"" Jul 31 18:45:18.183014 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.182975 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 18:45:18.183141 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.183022 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 18:45:18.183141 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.182982 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 18:45:18.223927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.223902 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rfnxc"] Jul 31 18:45:18.224730 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.224706 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda741de37_d1d8_4f52_b018_e8d38787f906.slice/crio-af0de735f6e55d7df267ba34bd88b7d83419c5134b09e25778df9a70df655f92 WatchSource:0}: Error finding container af0de735f6e55d7df267ba34bd88b7d83419c5134b09e25778df9a70df655f92: Status 404 returned error can't find the container with id af0de735f6e55d7df267ba34bd88b7d83419c5134b09e25778df9a70df655f92 Jul 31 18:45:18.234735 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.234707 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-skr92"] Jul 31 18:45:18.247519 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.246167 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.250378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.250356 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 31 18:45:18.256168 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.256147 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 31 18:45:18.256286 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.256268 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 31 18:45:18.256355 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.256342 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g86kn\"" Jul 31 18:45:18.262820 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.262801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-skr92"] Jul 31 18:45:18.290340 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgs9\" (UniqueName: \"kubernetes.io/projected/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-kube-api-access-sbgs9\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290340 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-textfile\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290580 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290360 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jb2\" (UniqueName: \"kubernetes.io/projected/0a524a1b-6de2-425f-84fc-5d7048f4983a-kube-api-access-m6jb2\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.290580 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.290580 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.290580 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a524a1b-6de2-425f-84fc-5d7048f4983a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-metrics-client-ca\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290635 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-root\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-tls\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-wtmp\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290748 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-sys\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.290957 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.290768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.342998 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.342957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6gr7q" event={"ID":"b178f3a3-fa58-4d04-9e33-bf0533dc0aea","Type":"ContainerStarted","Data":"193b37ad184e3236a3e3fcc32d2754e92560cc25cb9ee6c8119ef2552e9d86f1"} Jul 31 18:45:18.344077 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.344043 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nswnl" event={"ID":"82292e4a-5143-4fd1-8750-b8502a4041d6","Type":"ContainerStarted","Data":"f2e7de66cfd445bef3196f8b9c1f53b734147a2ae5118b0e64d4ecafafef951d"} Jul 31 18:45:18.345484 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.345460 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c987dcf5-7t58c" event={"ID":"793c842e-a5bc-4ed9-995d-1a8c9c325b2c","Type":"ContainerStarted","Data":"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a"} Jul 31 18:45:18.346721 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.346700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rfnxc" event={"ID":"a741de37-d1d8-4f52-b018-e8d38787f906","Type":"ContainerStarted","Data":"af0de735f6e55d7df267ba34bd88b7d83419c5134b09e25778df9a70df655f92"} Jul 31 18:45:18.368782 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.368725 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57c987dcf5-7t58c" podStartSLOduration=1.107130989 podStartE2EDuration="4.368706176s" totalImagesPullingTime="3.261575187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:18.368155978 +0000 UTC m=+66.906383629" watchObservedRunningTime="2026-07-31 18:45:18.368706176 +0000 UTC m=+66.906933827" Jul 31 18:45:18.391574 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgs9\" (UniqueName: \"kubernetes.io/projected/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-kube-api-access-sbgs9\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.391574 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.391765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-textfile\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.391765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jb2\" (UniqueName: \"kubernetes.io/projected/0a524a1b-6de2-425f-84fc-5d7048f4983a-kube-api-access-m6jb2\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.391765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.391860 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.391905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391857 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.391905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2663c406-3579-4757-96dc-3a79b46fdc2b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.391979 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391921 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvs7\" (UniqueName: \"kubernetes.io/projected/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-api-access-vlvs7\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.391979 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392058 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.391975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.392058 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a524a1b-6de2-425f-84fc-5d7048f4983a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.392058 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-metrics-client-ca\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392064 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392073 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-textfile\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-root\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-tls\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-root\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392158 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-wtmp\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392450 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-sys\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392450 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392213 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392450 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-wtmp\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392450 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-sys\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392680 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-metrics-client-ca\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.392740 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392683 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.393055 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.392906 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a524a1b-6de2-425f-84fc-5d7048f4983a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.394456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.394429 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.394674 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.394656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.394721 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.394687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a524a1b-6de2-425f-84fc-5d7048f4983a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.394759 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.394737 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-node-exporter-tls\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.402573 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.402549 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jb2\" (UniqueName: \"kubernetes.io/projected/0a524a1b-6de2-425f-84fc-5d7048f4983a-kube-api-access-m6jb2\") pod \"openshift-state-metrics-6bb4d7b4dd-8ltzf\" (UID: \"0a524a1b-6de2-425f-84fc-5d7048f4983a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.402677 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.402658 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgs9\" (UniqueName: \"kubernetes.io/projected/7245fb16-fdce-4d85-9f58-6c2dfc5dade6-kube-api-access-sbgs9\") pod \"node-exporter-ccppf\" (UID: \"7245fb16-fdce-4d85-9f58-6c2dfc5dade6\") " pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.465391 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.465362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" Jul 31 18:45:18.486611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.486536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccppf" Jul 31 18:45:18.492807 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492778 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.492928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.492928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492862 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2663c406-3579-4757-96dc-3a79b46fdc2b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.492928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvs7\" (UniqueName: \"kubernetes.io/projected/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-api-access-vlvs7\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.492928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.493127 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.492991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.493451 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.493388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2663c406-3579-4757-96dc-3a79b46fdc2b-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.493807 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.493784 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.494228 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.494201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.495682 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.495650 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.495682 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.495673 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.503231 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.503206 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvs7\" (UniqueName: \"kubernetes.io/projected/2663c406-3579-4757-96dc-3a79b46fdc2b-kube-api-access-vlvs7\") pod \"kube-state-metrics-6bd9664c4-skr92\" (UID: \"2663c406-3579-4757-96dc-3a79b46fdc2b\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.535191 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.535147 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7245fb16_fdce_4d85_9f58_6c2dfc5dade6.slice/crio-e510cf5008924c3fb8ad173b7522aedb69bf05e93a56a032497f50996b4d96d6 WatchSource:0}: Error finding container e510cf5008924c3fb8ad173b7522aedb69bf05e93a56a032497f50996b4d96d6: Status 404 returned error can't find the container with id e510cf5008924c3fb8ad173b7522aedb69bf05e93a56a032497f50996b4d96d6 Jul 31 18:45:18.556809 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.556782 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" Jul 31 18:45:18.641423 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.641393 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf"] Jul 31 18:45:18.644440 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.644400 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a524a1b_6de2_425f_84fc_5d7048f4983a.slice/crio-10c29915d23d1daa4a77776086cbd6efa76c29d467ece63bd4fbbffb85f272b5 WatchSource:0}: Error finding container 10c29915d23d1daa4a77776086cbd6efa76c29d467ece63bd4fbbffb85f272b5: Status 404 returned error can't find the container with id 10c29915d23d1daa4a77776086cbd6efa76c29d467ece63bd4fbbffb85f272b5 Jul 31 18:45:18.739697 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.739611 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-skr92"] Jul 31 18:45:18.746723 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:18.746675 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2663c406_3579_4757_96dc_3a79b46fdc2b.slice/crio-c86bda46a27b8bf6e2df680479a7ab9f83b0539156bbd4542036b111017be61d WatchSource:0}: Error finding container c86bda46a27b8bf6e2df680479a7ab9f83b0539156bbd4542036b111017be61d: Status 404 returned error can't find the container with id c86bda46a27b8bf6e2df680479a7ab9f83b0539156bbd4542036b111017be61d Jul 31 18:45:18.985173 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.985141 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:45:18.987559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.987342 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:18.996240 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:18.996111 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 18:45:19.006320 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.006255 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098624 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098666 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmx8x\" (UniqueName: \"kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098706 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.098990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.098869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx8x\" (UniqueName: \"kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200192 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200405 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200441 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.200472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.201056 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.201008 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.202360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.202317 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.202620 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.202596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.202726 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.202597 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.203689 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.203668 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.205436 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.205414 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.225429 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.225405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx8x\" (UniqueName: \"kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x\") pod \"console-7bd7cc4497-xbdjn\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.307772 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.307427 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:19.359049 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.358973 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccppf" event={"ID":"7245fb16-fdce-4d85-9f58-6c2dfc5dade6","Type":"ContainerStarted","Data":"e510cf5008924c3fb8ad173b7522aedb69bf05e93a56a032497f50996b4d96d6"} Jul 31 18:45:19.366510 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.366368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" event={"ID":"0a524a1b-6de2-425f-84fc-5d7048f4983a","Type":"ContainerStarted","Data":"eb0aef7afb39bea960700b4da8373eef4072c9ce42075d661fe81c5cebed93fd"} Jul 31 18:45:19.366510 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.366425 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" event={"ID":"0a524a1b-6de2-425f-84fc-5d7048f4983a","Type":"ContainerStarted","Data":"9316b2a8cce75df48936cf2ce67412ed922011d3439f58dfcf42f39f8dcbd0f8"} Jul 31 18:45:19.366510 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.366445 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" event={"ID":"0a524a1b-6de2-425f-84fc-5d7048f4983a","Type":"ContainerStarted","Data":"10c29915d23d1daa4a77776086cbd6efa76c29d467ece63bd4fbbffb85f272b5"} Jul 31 18:45:19.369263 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.369199 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" event={"ID":"2663c406-3579-4757-96dc-3a79b46fdc2b","Type":"ContainerStarted","Data":"c86bda46a27b8bf6e2df680479a7ab9f83b0539156bbd4542036b111017be61d"} Jul 31 18:45:19.390590 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.390542 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:45:19.398846 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.398814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.408124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.407178 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 18:45:19.408124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.407447 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 18:45:19.408124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.407642 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 18:45:19.408124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.407817 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 18:45:19.408124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.407992 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h76gt\"" Jul 31 18:45:19.410860 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.410837 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:45:19.412250 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.412231 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 18:45:19.412786 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.412759 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 18:45:19.412939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.412651 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 18:45:19.413030 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.412694 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 18:45:19.413263 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.412558 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 18:45:19.496467 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.496381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504120 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504285 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cwm\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.504601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.504349 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.506348 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.505420 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.506348 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.505471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.506348 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.505523 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.606929 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.606976 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607004 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cwm\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607113 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.607301 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.609424 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.609424 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.607362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.609629 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.609601 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.609914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.609892 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.611526 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.611188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.615838 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.615796 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.618440 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.618412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.619637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.619587 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cwm\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.619719 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.619634 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.619878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.619838 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.619984 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.619948 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.621091 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.621063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.622178 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.622130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.624087 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.624061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.624613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.624573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:19.718124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:19.717847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:45:20.374812 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:20.374723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd7cc4497-xbdjn" event={"ID":"778f32d5-680d-42f9-8ee8-bfd5f073045f","Type":"ContainerStarted","Data":"49af0fa9210a9fdd37efd479c97228ff4eba037441b2139d5120d18538833e58"} Jul 31 18:45:21.212538 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.212417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:45:21.372278 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.372189 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b974ff869-jj6t6"] Jul 31 18:45:21.376845 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.376420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.380876 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mmnzd\"" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.381023 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.381036 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.381105 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.381023 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 31 18:45:21.381378 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.381313 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 31 18:45:21.382202 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.382041 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7kmh33g987jn3\"" Jul 31 18:45:21.391526 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.391466 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b974ff869-jj6t6"] Jul 31 18:45:21.533759 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-grpc-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.533939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533777 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.533939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.533939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533884 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.533939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.533939 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533926 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-metrics-client-ca\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.534186 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.533955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dh78\" (UniqueName: \"kubernetes.io/projected/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-kube-api-access-6dh78\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.534186 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.534039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-metrics-client-ca\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dh78\" (UniqueName: \"kubernetes.io/projected/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-kube-api-access-6dh78\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-grpc-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.635684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.635468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.636172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.636145 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-metrics-client-ca\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.638728 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.638700 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.638920 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.638894 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.639027 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.638982 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.639466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.639441 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.639466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.639454 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-grpc-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.642206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.642158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-secret-thanos-querier-tls\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.646572 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.646539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dh78\" (UniqueName: \"kubernetes.io/projected/6cc6b4fd-55fa-4c1b-a360-e20775d90b63-kube-api-access-6dh78\") pod \"thanos-querier-b974ff869-jj6t6\" (UID: \"6cc6b4fd-55fa-4c1b-a360-e20775d90b63\") " pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:21.688718 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:21.688677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:22.081359 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:22.081325 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12508d35_c009_41e8_865e_42fbaab02b2b.slice/crio-42bc4af96afcae2286c725539ecdc02213381bf4dfaed0fd87375d1324150418 WatchSource:0}: Error finding container 42bc4af96afcae2286c725539ecdc02213381bf4dfaed0fd87375d1324150418: Status 404 returned error can't find the container with id 42bc4af96afcae2286c725539ecdc02213381bf4dfaed0fd87375d1324150418 Jul 31 18:45:22.386087 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.386009 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"42bc4af96afcae2286c725539ecdc02213381bf4dfaed0fd87375d1324150418"} Jul 31 18:45:22.567077 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.567044 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bdcd74c68-vfbcj"] Jul 31 18:45:22.569433 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.569410 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.575900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.575526 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 31 18:45:22.575900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.575545 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-22bfco0fdilfc\"" Jul 31 18:45:22.575900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.575556 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 31 18:45:22.575900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.575572 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jlfjb\"" Jul 31 18:45:22.577628 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.577606 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 31 18:45:22.578397 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.578357 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 31 18:45:22.588835 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.588811 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bdcd74c68-vfbcj"] Jul 31 18:45:22.647179 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647146 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-tls\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac265491-b08e-4d17-8512-3060b343591b-audit-log\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-metrics-server-audit-profiles\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647357 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-client-certs\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-client-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.647548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.647537 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bqp\" (UniqueName: \"kubernetes.io/projected/ac265491-b08e-4d17-8512-3060b343591b-kube-api-access-99bqp\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.748864 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-tls\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.748864 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac265491-b08e-4d17-8512-3060b343591b-audit-log\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748897 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-metrics-server-audit-profiles\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748953 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-client-certs\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.748991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-client-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.749033 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bqp\" (UniqueName: \"kubernetes.io/projected/ac265491-b08e-4d17-8512-3060b343591b-kube-api-access-99bqp\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749368 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.749300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac265491-b08e-4d17-8512-3060b343591b-audit-log\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.749764 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.749735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.750257 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.750238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac265491-b08e-4d17-8512-3060b343591b-metrics-server-audit-profiles\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.751762 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.751738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-tls\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.751864 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.751836 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-secret-metrics-server-client-certs\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.752104 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.752088 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac265491-b08e-4d17-8512-3060b343591b-client-ca-bundle\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.762284 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.762267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bqp\" (UniqueName: \"kubernetes.io/projected/ac265491-b08e-4d17-8512-3060b343591b-kube-api-access-99bqp\") pod \"metrics-server-7bdcd74c68-vfbcj\" (UID: \"ac265491-b08e-4d17-8512-3060b343591b\") " pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.880639 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.880604 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:22.922711 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.922628 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng"] Jul 31 18:45:22.927172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.927142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:22.930216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.930196 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 31 18:45:22.930366 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.930196 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wrr62\"" Jul 31 18:45:22.934469 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:22.934447 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng"] Jul 31 18:45:23.052202 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.052168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5140d648-06e5-4177-a382-9fac12613660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.052369 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.052239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrs2\" (UniqueName: \"kubernetes.io/projected/5140d648-06e5-4177-a382-9fac12613660-kube-api-access-kgrs2\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.152992 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.152950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5140d648-06e5-4177-a382-9fac12613660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.153171 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.153069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrs2\" (UniqueName: \"kubernetes.io/projected/5140d648-06e5-4177-a382-9fac12613660-kube-api-access-kgrs2\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.156024 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.155994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5140d648-06e5-4177-a382-9fac12613660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.163375 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.163351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrs2\" (UniqueName: \"kubernetes.io/projected/5140d648-06e5-4177-a382-9fac12613660-kube-api-access-kgrs2\") pod \"monitoring-plugin-6dffbbd67-t8jng\" (UID: \"5140d648-06e5-4177-a382-9fac12613660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.220100 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.220013 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:23.239136 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.239082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:23.259006 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.258979 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:23.261353 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.261334 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.275475 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.275414 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:23.355481 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xjz\" (UniqueName: \"kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355683 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355683 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355562 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355683 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355683 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355683 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.355911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.355698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.456469 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.456435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.456469 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.456472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.456988 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.456523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.456988 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.456560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.456988 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.456606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.457142 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.457060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.457142 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.457130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xjz\" (UniqueName: \"kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.457457 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.457367 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.457927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.457530 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.457927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.457668 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.458076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.458053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.459544 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.459522 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.459637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.459573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.463624 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.463601 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d5b9968c5-h9rhr"] Jul 31 18:45:23.467530 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.467513 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.471823 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.471773 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 31 18:45:23.472143 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.472118 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 31 18:45:23.472244 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.472165 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 31 18:45:23.472345 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.472328 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7qhnd\"" Jul 31 18:45:23.472465 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.472446 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 31 18:45:23.472571 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.472556 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 31 18:45:23.477973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.477955 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 31 18:45:23.483714 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.483689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xjz\" (UniqueName: \"kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz\") pod \"console-6bdc5474d-l8rs4\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.486317 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.486294 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5b9968c5-h9rhr"] Jul 31 18:45:23.557616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557583 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-federate-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-metrics-client-ca\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557672 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557863 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-serving-certs-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.557911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.557907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njt5l\" (UniqueName: \"kubernetes.io/projected/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-kube-api-access-njt5l\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.572190 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.572148 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:23.661202 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661314 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661380 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-serving-certs-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661440 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661385 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njt5l\" (UniqueName: \"kubernetes.io/projected/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-kube-api-access-njt5l\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661440 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661425 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-federate-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661595 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-metrics-client-ca\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661595 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.661595 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.661570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.663054 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.663028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.666453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.666427 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-serving-certs-ca-bundle\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.667092 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.667068 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-metrics-client-ca\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.678829 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.673305 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-telemeter-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.678829 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.677783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.686268 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.685955 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-federate-client-tls\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.692048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.691975 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njt5l\" (UniqueName: \"kubernetes.io/projected/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-kube-api-access-njt5l\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.692709 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.692675 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccaa60f-50b0-4cf6-9fcd-8974586dfe18-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5b9968c5-h9rhr\" (UID: \"cccaa60f-50b0-4cf6-9fcd-8974586dfe18\") " pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.793287 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.787360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" Jul 31 18:45:23.825765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:23.825650 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng"] Jul 31 18:45:24.079693 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.079094 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5b9968c5-h9rhr"] Jul 31 18:45:24.084266 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:24.084234 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccaa60f_50b0_4cf6_9fcd_8974586dfe18.slice/crio-41101376ab4985ecf02b001b16239f78cb42e3cfa533bb4e8713e218f1924ab0 WatchSource:0}: Error finding container 41101376ab4985ecf02b001b16239f78cb42e3cfa533bb4e8713e218f1924ab0: Status 404 returned error can't find the container with id 41101376ab4985ecf02b001b16239f78cb42e3cfa533bb4e8713e218f1924ab0 Jul 31 18:45:24.100324 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.100222 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b974ff869-jj6t6"] Jul 31 18:45:24.108797 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.108770 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:24.112210 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:24.112176 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc6b4fd_55fa_4c1b_a360_e20775d90b63.slice/crio-1abda7cce6378852047cad03b552624bc93e3d7ddbf9ec0d5765943f9b149770 WatchSource:0}: Error finding container 1abda7cce6378852047cad03b552624bc93e3d7ddbf9ec0d5765943f9b149770: Status 404 returned error can't find the container with id 1abda7cce6378852047cad03b552624bc93e3d7ddbf9ec0d5765943f9b149770 Jul 31 18:45:24.113185 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:24.113157 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d425f4a_e7cd_492c_9913_bb8033384f79.slice/crio-5955c023ec2203dbc64de9562f593805906efc9ed43f8e1d84d0a5c2466b0cba WatchSource:0}: Error finding container 5955c023ec2203dbc64de9562f593805906efc9ed43f8e1d84d0a5c2466b0cba: Status 404 returned error can't find the container with id 5955c023ec2203dbc64de9562f593805906efc9ed43f8e1d84d0a5c2466b0cba Jul 31 18:45:24.136346 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.136261 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bdcd74c68-vfbcj"] Jul 31 18:45:24.144150 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:45:24.144118 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac265491_b08e_4d17_8512_3060b343591b.slice/crio-974607d9ce3e01a644c6077a55351e285933a8d980d3ebbf1f69cffcf1f37e79 WatchSource:0}: Error finding container 974607d9ce3e01a644c6077a55351e285933a8d980d3ebbf1f69cffcf1f37e79: Status 404 returned error can't find the container with id 974607d9ce3e01a644c6077a55351e285933a8d980d3ebbf1f69cffcf1f37e79 Jul 31 18:45:24.398096 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.398051 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"1abda7cce6378852047cad03b552624bc93e3d7ddbf9ec0d5765943f9b149770"} Jul 31 18:45:24.400391 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.400334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" event={"ID":"0a524a1b-6de2-425f-84fc-5d7048f4983a","Type":"ContainerStarted","Data":"77ef078e3ea8e530aa2b33f463e6198b9270188f4b3cbed38f9a057e1829344a"} Jul 31 18:45:24.402246 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.402218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd7cc4497-xbdjn" event={"ID":"778f32d5-680d-42f9-8ee8-bfd5f073045f","Type":"ContainerStarted","Data":"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2"} Jul 31 18:45:24.404121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.403759 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rfnxc" event={"ID":"a741de37-d1d8-4f52-b018-e8d38787f906","Type":"ContainerStarted","Data":"161300136b9be5bfcb162781a996e642d102acfeba91beba4a906d6385abd907"} Jul 31 18:45:24.404121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.403803 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:24.406101 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.406063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" event={"ID":"2663c406-3579-4757-96dc-3a79b46fdc2b","Type":"ContainerStarted","Data":"920770c481a343b996efb3d86d8a4e794ab152c64e4b2405787824d97b7c34c9"} Jul 31 18:45:24.406206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.406113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" event={"ID":"2663c406-3579-4757-96dc-3a79b46fdc2b","Type":"ContainerStarted","Data":"810b4aacf6278a9fac5ecd25176d7b4a036fb52bf9ea42e504ee3bee80d7d3dc"} Jul 31 18:45:24.406206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.406140 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" event={"ID":"2663c406-3579-4757-96dc-3a79b46fdc2b","Type":"ContainerStarted","Data":"ba9c65d800210bd5e85f0f59c7fa974d2c7599cd14a7993ec5ff2281b1eefaa3"} Jul 31 18:45:24.408379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.408023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc5474d-l8rs4" event={"ID":"4d425f4a-e7cd-492c-9913-bb8033384f79","Type":"ContainerStarted","Data":"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29"} Jul 31 18:45:24.408379 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.408053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc5474d-l8rs4" event={"ID":"4d425f4a-e7cd-492c-9913-bb8033384f79","Type":"ContainerStarted","Data":"5955c023ec2203dbc64de9562f593805906efc9ed43f8e1d84d0a5c2466b0cba"} Jul 31 18:45:24.409102 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.409075 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" event={"ID":"5140d648-06e5-4177-a382-9fac12613660","Type":"ContainerStarted","Data":"b9bd4a79aff43d6608ab1b409b1324f6b6392193e1633c28771bd7f4f28e7656"} Jul 31 18:45:24.411956 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.411911 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6gr7q" event={"ID":"b178f3a3-fa58-4d04-9e33-bf0533dc0aea","Type":"ContainerStarted","Data":"36fda9fe4a7224dbc94647715350b077558a5cc246ff7f3026d2a55e0647b5f3"} Jul 31 18:45:24.414404 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.414382 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nswnl" event={"ID":"82292e4a-5143-4fd1-8750-b8502a4041d6","Type":"ContainerStarted","Data":"e7a941d0bd20c82c00d77dbc2f27c09a4595cf3a15622d699650be8b7ae3f7db"} Jul 31 18:45:24.414520 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.414427 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nswnl" event={"ID":"82292e4a-5143-4fd1-8750-b8502a4041d6","Type":"ContainerStarted","Data":"954948f316c8f56352eafb47c02573c538f338efab5f62dd299cb03eb29c6f2e"} Jul 31 18:45:24.416082 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.415995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" event={"ID":"cccaa60f-50b0-4cf6-9fcd-8974586dfe18","Type":"ContainerStarted","Data":"41101376ab4985ecf02b001b16239f78cb42e3cfa533bb4e8713e218f1924ab0"} Jul 31 18:45:24.417546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.417523 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" event={"ID":"ac265491-b08e-4d17-8512-3060b343591b","Type":"ContainerStarted","Data":"974607d9ce3e01a644c6077a55351e285933a8d980d3ebbf1f69cffcf1f37e79"} Jul 31 18:45:24.419166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.419140 2566 generic.go:356] "Generic (PLEG): container finished" podID="7245fb16-fdce-4d85-9f58-6c2dfc5dade6" containerID="61f7d3e6d16a3409182a2503a3a487d9046e1d2fd44625f2b09ac5e87a9dbbe1" exitCode=0 Jul 31 18:45:24.419263 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.419189 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccppf" event={"ID":"7245fb16-fdce-4d85-9f58-6c2dfc5dade6","Type":"ContainerDied","Data":"61f7d3e6d16a3409182a2503a3a487d9046e1d2fd44625f2b09ac5e87a9dbbe1"} Jul 31 18:45:24.478370 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.478308 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-8ltzf" podStartSLOduration=1.811363469 podStartE2EDuration="6.478292008s" totalImagesPullingTime="4.666928539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.450012393 +0000 UTC m=+72.988240042" watchObservedRunningTime="2026-07-31 18:45:24.478292008 +0000 UTC m=+73.016519637" Jul 31 18:45:24.519121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.519031 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:24.523247 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.523186 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd7cc4497-xbdjn" podStartSLOduration=6.523170681 podStartE2EDuration="6.523170681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.52305852 +0000 UTC m=+73.061286172" watchObservedRunningTime="2026-07-31 18:45:24.523170681 +0000 UTC m=+73.061398332" Jul 31 18:45:24.554585 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.554432 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-skr92" podStartSLOduration=1.734956585 podStartE2EDuration="6.554410391s" totalImagesPullingTime="4.819453806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.552770033 +0000 UTC m=+73.090997694" watchObservedRunningTime="2026-07-31 18:45:24.554410391 +0000 UTC m=+73.092638043" Jul 31 18:45:24.575953 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.575899 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rfnxc" podStartSLOduration=67.212742473 podStartE2EDuration="1m12.575882526s" totalImagesPullingTime="5.363140053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.575387608 +0000 UTC m=+73.113615260" watchObservedRunningTime="2026-07-31 18:45:24.575882526 +0000 UTC m=+73.114110177" Jul 31 18:45:24.599629 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.599573 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bdc5474d-l8rs4" podStartSLOduration=1.5995567560000001 podStartE2EDuration="1.599556756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.598138919 +0000 UTC m=+73.136366570" watchObservedRunningTime="2026-07-31 18:45:24.599556756 +0000 UTC m=+73.137784385" Jul 31 18:45:24.625331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.625271 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nswnl" podStartSLOduration=67.106323549 podStartE2EDuration="1m12.625249016s" totalImagesPullingTime="5.518925467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.623357319 +0000 UTC m=+73.161584982" watchObservedRunningTime="2026-07-31 18:45:24.625249016 +0000 UTC m=+73.163476666" Jul 31 18:45:24.649998 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.649229 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:45:24.653917 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.653886 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.661506 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.661314 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662017 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662047 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662124 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662246 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662268 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662436 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b88jmc869aguu\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662591 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.662764 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.663438 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-npvzd\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.663855 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 18:45:24.664243 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.663987 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 18:45:24.665089 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.664961 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 18:45:24.665227 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.665202 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 18:45:24.666518 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.666255 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 18:45:24.669624 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.669604 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 18:45:24.680655 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.680592 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6gr7q" podStartSLOduration=67.142114678 podStartE2EDuration="1m12.680569018s" totalImagesPullingTime="5.53845434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.663671908 +0000 UTC m=+73.201899560" watchObservedRunningTime="2026-07-31 18:45:24.680569018 +0000 UTC m=+73.218796669" Jul 31 18:45:24.682126 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.682099 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:45:24.790479 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790620 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790522 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790620 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790655 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790767 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.790878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.790824 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791072 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6qq\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.791529 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.791411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892238 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892323 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892505 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892588 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6qq\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892629 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892649 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.892914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.893769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892717 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.893769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.893769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.892795 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.893769 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.893547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.894674 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.894634 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.894798 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.894710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.896550 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.896020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.897837 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.897389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.897837 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.897520 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.898013 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.897855 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.900372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.899897 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.900372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.900340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.900589 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.900566 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.901456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.901398 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.902455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.902431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.902763 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.902745 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.903784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.903767 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.904201 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.904180 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.904583 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.904561 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.905395 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.904027 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.907791 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.907766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.908704 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.908660 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.922838 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.922808 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6qq\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq\") pod \"prometheus-k8s-0\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:24.967265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:24.967015 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:25.199878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.197930 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:45:25.427526 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.427192 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" exitCode=0 Jul 31 18:45:25.427526 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.427327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} Jul 31 18:45:25.427526 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.427393 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"aefc3558b526071ef2e24f489b9e80ffcf5bfda8109217944f2590db68a9ead8"} Jul 31 18:45:25.431606 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.431280 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccppf" event={"ID":"7245fb16-fdce-4d85-9f58-6c2dfc5dade6","Type":"ContainerStarted","Data":"a16dcd43c490d35a3b60c7f40f662892407b79497127a5253a05b049f74b5e9d"} Jul 31 18:45:25.431606 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.431336 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccppf" event={"ID":"7245fb16-fdce-4d85-9f58-6c2dfc5dade6","Type":"ContainerStarted","Data":"1601119be6d2f806e43b08ae0ee0c7aea82767a6dbc4834d906d7fbbff1bc52f"} Jul 31 18:45:25.436730 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.434178 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684" exitCode=0 Jul 31 18:45:25.436730 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.435308 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684"} Jul 31 18:45:25.552225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:25.552105 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ccppf" podStartSLOduration=2.346994365 podStartE2EDuration="7.552081069s" totalImagesPullingTime="5.032412513s" totalInitContainerRuntime="172.674191ms" isStatefulPod=true podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:25.550778466 +0000 UTC m=+74.089006119" watchObservedRunningTime="2026-07-31 18:45:25.552081069 +0000 UTC m=+74.090308722" Jul 31 18:45:29.308230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:29.308195 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:29.308595 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:29.308295 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:29.313064 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:29.313046 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:29.459676 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:29.459652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:45:30.460757 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.460716 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" event={"ID":"cccaa60f-50b0-4cf6-9fcd-8974586dfe18","Type":"ContainerStarted","Data":"437993f3fa4f49c56417d5007e0906bec620063a887e60254aab09ea1a84130d"} Jul 31 18:45:30.461237 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.460790 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" event={"ID":"cccaa60f-50b0-4cf6-9fcd-8974586dfe18","Type":"ContainerStarted","Data":"5a33b7c50793a5ab88de131140ffa41a08ccfab82692944f6a10ffad2a1ed406"} Jul 31 18:45:30.461237 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.460824 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" event={"ID":"cccaa60f-50b0-4cf6-9fcd-8974586dfe18","Type":"ContainerStarted","Data":"04038f1ff0c754151f1febe004c314eb2cb5914a2a2210662a2aa66b16d56816"} Jul 31 18:45:30.462670 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.462638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" event={"ID":"ac265491-b08e-4d17-8512-3060b343591b","Type":"ContainerStarted","Data":"ebec85546a28fab0cc2a18df650347cb9a22882d39931241ede03b4e9b762593"} Jul 31 18:45:30.466652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.466373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb"} Jul 31 18:45:30.466652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.466441 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34"} Jul 31 18:45:30.466652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.466479 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649"} Jul 31 18:45:30.466652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.466534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd"} Jul 31 18:45:30.466652 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.466573 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157"} Jul 31 18:45:30.468773 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.468751 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"e9f03a946816d9191be25d1c59d65f23015885822537c33c9151aa46946ad928"} Jul 31 18:45:30.468865 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.468813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"3e196b9b1bc986bfa82573e739e9662a6b71a17356600d84bf3ec3f6c987bd0f"} Jul 31 18:45:30.468920 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.468865 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"ddc5b0fc0d922e9c4d6fdc55d76b41e4b8d6f100f1b60c1bc7ca63aae2184c7c"} Jul 31 18:45:30.470416 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.470386 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" event={"ID":"5140d648-06e5-4177-a382-9fac12613660","Type":"ContainerStarted","Data":"e16d028873a924f07ed2e10a35b7f9f927ceb6872c8ad739fe16e01e3224e4d2"} Jul 31 18:45:30.470547 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.470527 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:30.474356 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} Jul 31 18:45:30.474443 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474385 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} Jul 31 18:45:30.474443 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} Jul 31 18:45:30.474443 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} Jul 31 18:45:30.474613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} Jul 31 18:45:30.474613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.474525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerStarted","Data":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} Jul 31 18:45:30.476512 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.476481 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" Jul 31 18:45:30.496926 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.496863 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d5b9968c5-h9rhr" podStartSLOduration=2.2538998279999998 podStartE2EDuration="7.496846432s" totalImagesPullingTime="5.242946604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:30.496284776 +0000 UTC m=+79.034512452" watchObservedRunningTime="2026-07-31 18:45:30.496846432 +0000 UTC m=+79.035074083" Jul 31 18:45:30.519052 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.518996 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t8jng" podStartSLOduration=3.058651469 podStartE2EDuration="8.518979046s" totalImagesPullingTime="5.460327577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:30.518345707 +0000 UTC m=+79.056573358" watchObservedRunningTime="2026-07-31 18:45:30.518979046 +0000 UTC m=+79.057206674" Jul 31 18:45:30.552015 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.551938 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.616386248 podStartE2EDuration="6.551915735s" totalImagesPullingTime="3.918643662s" totalInitContainerRuntime="16.885825ms" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:30.550425982 +0000 UTC m=+79.088653636" watchObservedRunningTime="2026-07-31 18:45:30.551915735 +0000 UTC m=+79.090143391" Jul 31 18:45:30.573783 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.573735 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" podStartSLOduration=3.391447903 podStartE2EDuration="8.573717536s" totalImagesPullingTime="5.182269633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:30.5722391 +0000 UTC m=+79.110466753" watchObservedRunningTime="2026-07-31 18:45:30.573717536 +0000 UTC m=+79.111945184" Jul 31 18:45:30.600946 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:30.600919 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:31.487508 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.487461 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerStarted","Data":"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917"} Jul 31 18:45:31.490646 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.490608 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"a30cd720551fea3811cd9b5d2cc5f53dedbd5fe8632cfcf25b5d195638d3173f"} Jul 31 18:45:31.490751 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.490690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"ffae3c49e5da8b169b2a2d2462e0a92cdc4c83b29abea1d10498d2467f8403b7"} Jul 31 18:45:31.490751 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.490722 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" event={"ID":"6cc6b4fd-55fa-4c1b-a360-e20775d90b63","Type":"ContainerStarted","Data":"a7febee98e52226201f1695df1fa2fc78db6227aa7ddf85a6ec726be295d20b3"} Jul 31 18:45:31.491405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.491385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:31.523991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.523691 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.047590679 podStartE2EDuration="12.523646389s" totalImagesPullingTime="7.453312356s" totalInitContainerRuntime="22.743354ms" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:31.521687212 +0000 UTC m=+80.059914863" watchObservedRunningTime="2026-07-31 18:45:31.523646389 +0000 UTC m=+80.061874041" Jul 31 18:45:31.548400 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:31.548349 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" podStartSLOduration=4.456723276 podStartE2EDuration="10.54833206s" totalImagesPullingTime="6.091608784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:31.547549277 +0000 UTC m=+80.085776945" watchObservedRunningTime="2026-07-31 18:45:31.54833206 +0000 UTC m=+80.086559710" Jul 31 18:45:33.573247 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:33.573110 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:34.967544 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:34.967468 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:45:37.501137 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:37.501106 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b974ff869-jj6t6" Jul 31 18:45:41.564133 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:41.564101 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:45:42.880711 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:42.880672 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:42.881172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:42.880731 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:45:48.241178 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.241110 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57c987dcf5-7t58c" podUID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" containerName="console" containerID="cri-o://7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a" gracePeriod=15 Jul 31 18:45:48.486900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.486878 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:48.575225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.575148 2566 generic.go:356] "Generic (PLEG): container finished" podID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" containerID="7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a" exitCode=0 Jul 31 18:45:48.575225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.575188 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c987dcf5-7t58c" event={"ID":"793c842e-a5bc-4ed9-995d-1a8c9c325b2c","Type":"ContainerDied","Data":"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a"} Jul 31 18:45:48.575422 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.575222 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c987dcf5-7t58c" Jul 31 18:45:48.575422 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.575256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c987dcf5-7t58c" event={"ID":"793c842e-a5bc-4ed9-995d-1a8c9c325b2c","Type":"ContainerDied","Data":"ec70555cd74da7403f2f83858843bcc1edee2e3fec69317b7ec6e13d6bf30b1d"} Jul 31 18:45:48.575422 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.575290 2566 scope.go:117] "RemoveContainer" containerID="7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a" Jul 31 18:45:48.583877 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.583860 2566 scope.go:117] "RemoveContainer" containerID="7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a" Jul 31 18:45:48.584134 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:48.584116 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a\": container with ID starting with 7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a not found: ID does not exist" containerID="7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a" Jul 31 18:45:48.584183 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.584146 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a"} err="failed to get container status \"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a\": rpc error: code = NotFound desc = could not find container \"7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a\": container with ID starting with 7ea57a4288454fd8e1712f45aeed9dda0ce0c9aa89dc556914a042325c34c72a not found: ID does not exist" Jul 31 18:45:48.650981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.650947 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651111 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651052 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651111 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651181 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651127 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651181 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651161 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651253 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651197 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px4zd\" (UniqueName: \"kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd\") pod \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\" (UID: \"793c842e-a5bc-4ed9-995d-1a8c9c325b2c\") " Jul 31 18:45:48.651417 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651387 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:48.651509 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651470 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-oauth-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.651509 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651472 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca" (OuterVolumeSpecName: "service-ca") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:48.651614 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.651515 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config" (OuterVolumeSpecName: "console-config") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:48.653588 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.653564 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:48.653690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.653619 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:48.653690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.653610 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd" (OuterVolumeSpecName: "kube-api-access-px4zd") pod "793c842e-a5bc-4ed9-995d-1a8c9c325b2c" (UID: "793c842e-a5bc-4ed9-995d-1a8c9c325b2c"). InnerVolumeSpecName "kube-api-access-px4zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:45:48.752675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.752638 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-service-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.752675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.752668 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-oauth-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.752675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.752678 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.752675 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.752686 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-console-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.752941 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.752695 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px4zd\" (UniqueName: \"kubernetes.io/projected/793c842e-a5bc-4ed9-995d-1a8c9c325b2c-kube-api-access-px4zd\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:48.908265 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.905589 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:48.910283 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:48.910262 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57c987dcf5-7t58c"] Jul 31 18:45:50.034368 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:50.034335 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" path="/var/lib/kubelet/pods/793c842e-a5bc-4ed9-995d-1a8c9c325b2c/volumes" Jul 31 18:45:55.437725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.437695 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rfnxc" Jul 31 18:45:55.627524 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.627463 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bdc5474d-l8rs4" podUID="4d425f4a-e7cd-492c-9913-bb8033384f79" containerName="console" containerID="cri-o://db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29" gracePeriod=15 Jul 31 18:45:55.913673 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.913648 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:55.922181 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922161 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58xjz\" (UniqueName: \"kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922203 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922225 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922245 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922460 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922315 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922460 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922369 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922460 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922396 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert\") pod \"4d425f4a-e7cd-492c-9913-bb8033384f79\" (UID: \"4d425f4a-e7cd-492c-9913-bb8033384f79\") " Jul 31 18:45:55.922691 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922661 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:55.922825 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922688 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config" (OuterVolumeSpecName: "console-config") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:55.922825 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922757 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:55.922971 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.922846 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:45:55.924566 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.924541 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:55.925022 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.925003 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz" (OuterVolumeSpecName: "kube-api-access-58xjz") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "kube-api-access-58xjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:45:55.925079 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:55.925018 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d425f4a-e7cd-492c-9913-bb8033384f79" (UID: "4d425f4a-e7cd-492c-9913-bb8033384f79"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023270 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xjz\" (UniqueName: \"kubernetes.io/projected/4d425f4a-e7cd-492c-9913-bb8033384f79-kube-api-access-58xjz\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023306 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-console-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023316 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-oauth-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023324 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d425f4a-e7cd-492c-9913-bb8033384f79-console-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023333 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-service-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023342 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-trusted-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.023358 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.023351 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d425f4a-e7cd-492c-9913-bb8033384f79-oauth-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:45:56.609585 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.609548 2566 generic.go:356] "Generic (PLEG): container finished" podID="4d425f4a-e7cd-492c-9913-bb8033384f79" containerID="db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29" exitCode=0 Jul 31 18:45:56.610031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.609595 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc5474d-l8rs4" Jul 31 18:45:56.610031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.609642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc5474d-l8rs4" event={"ID":"4d425f4a-e7cd-492c-9913-bb8033384f79","Type":"ContainerDied","Data":"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29"} Jul 31 18:45:56.610031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.609714 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc5474d-l8rs4" event={"ID":"4d425f4a-e7cd-492c-9913-bb8033384f79","Type":"ContainerDied","Data":"5955c023ec2203dbc64de9562f593805906efc9ed43f8e1d84d0a5c2466b0cba"} Jul 31 18:45:56.610031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.609747 2566 scope.go:117] "RemoveContainer" containerID="db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29" Jul 31 18:45:56.618036 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.618018 2566 scope.go:117] "RemoveContainer" containerID="db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29" Jul 31 18:45:56.618325 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:45:56.618306 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29\": container with ID starting with db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29 not found: ID does not exist" containerID="db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29" Jul 31 18:45:56.618373 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.618335 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29"} err="failed to get container status \"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29\": rpc error: code = NotFound desc = could not find container \"db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29\": container with ID starting with db025eb1afd54712329f821c49135d72ebd1e9a0ed238c29d5e035a66dfa7a29 not found: ID does not exist" Jul 31 18:45:56.633037 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.633010 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:56.645502 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:56.645460 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bdc5474d-l8rs4"] Jul 31 18:45:58.034312 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:45:58.034279 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d425f4a-e7cd-492c-9913-bb8033384f79" path="/var/lib/kubelet/pods/4d425f4a-e7cd-492c-9913-bb8033384f79/volumes" Jul 31 18:46:02.886169 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:02.886137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:46:02.890038 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:02.890012 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bdcd74c68-vfbcj" Jul 31 18:46:06.584154 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.584095 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bd7cc4497-xbdjn" podUID="778f32d5-680d-42f9-8ee8-bfd5f073045f" containerName="console" containerID="cri-o://4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2" gracePeriod=15 Jul 31 18:46:06.835911 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.835834 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:46:06.916331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916292 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916535 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916378 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916535 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916417 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916535 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916443 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916535 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmx8x\" (UniqueName: \"kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916741 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916537 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916741 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916593 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert\") pod \"778f32d5-680d-42f9-8ee8-bfd5f073045f\" (UID: \"778f32d5-680d-42f9-8ee8-bfd5f073045f\") " Jul 31 18:46:06.916872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916843 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config" (OuterVolumeSpecName: "console-config") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:06.916978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916893 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca" (OuterVolumeSpecName: "service-ca") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:06.916978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.916913 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:06.917351 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.917328 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:06.918879 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.918843 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:06.918879 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.918851 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x" (OuterVolumeSpecName: "kube-api-access-bmx8x") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "kube-api-access-bmx8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:46:06.919014 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:06.918915 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "778f32d5-680d-42f9-8ee8-bfd5f073045f" (UID: "778f32d5-680d-42f9-8ee8-bfd5f073045f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:07.017260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017217 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-oauth-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017252 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017261 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-trusted-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017269 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-service-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017558 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017278 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmx8x\" (UniqueName: \"kubernetes.io/projected/778f32d5-680d-42f9-8ee8-bfd5f073045f-kube-api-access-bmx8x\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017558 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017286 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/778f32d5-680d-42f9-8ee8-bfd5f073045f-console-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.017558 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.017294 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/778f32d5-680d-42f9-8ee8-bfd5f073045f-oauth-serving-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:07.658834 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.658802 2566 generic.go:356] "Generic (PLEG): container finished" podID="778f32d5-680d-42f9-8ee8-bfd5f073045f" containerID="4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2" exitCode=0 Jul 31 18:46:07.659297 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.658854 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd7cc4497-xbdjn" Jul 31 18:46:07.659297 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.658850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd7cc4497-xbdjn" event={"ID":"778f32d5-680d-42f9-8ee8-bfd5f073045f","Type":"ContainerDied","Data":"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2"} Jul 31 18:46:07.659297 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.658969 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd7cc4497-xbdjn" event={"ID":"778f32d5-680d-42f9-8ee8-bfd5f073045f","Type":"ContainerDied","Data":"49af0fa9210a9fdd37efd479c97228ff4eba037441b2139d5120d18538833e58"} Jul 31 18:46:07.659297 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.658994 2566 scope.go:117] "RemoveContainer" containerID="4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2" Jul 31 18:46:07.669691 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.669667 2566 scope.go:117] "RemoveContainer" containerID="4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2" Jul 31 18:46:07.669972 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:07.669954 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2\": container with ID starting with 4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2 not found: ID does not exist" containerID="4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2" Jul 31 18:46:07.670038 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.669985 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2"} err="failed to get container status \"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2\": rpc error: code = NotFound desc = could not find container \"4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2\": container with ID starting with 4cdce85ac774b3baf70a3b9ec00d92d796a949e0095f2bff71ddf3a03634aaf2 not found: ID does not exist" Jul 31 18:46:07.693193 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.692665 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:46:07.695135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:07.695114 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd7cc4497-xbdjn"] Jul 31 18:46:08.034456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:08.034375 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778f32d5-680d-42f9-8ee8-bfd5f073045f" path="/var/lib/kubelet/pods/778f32d5-680d-42f9-8ee8-bfd5f073045f/volumes" Jul 31 18:46:24.967977 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:24.967928 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:24.988307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:24.988278 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:25.747171 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:25.747143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:38.672697 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.672664 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.672953 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="alertmanager" containerID="cri-o://bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157" gracePeriod=120 Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.673027 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-metric" containerID="cri-o://169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb" gracePeriod=120 Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.673050 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="prom-label-proxy" containerID="cri-o://3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917" gracePeriod=120 Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.673098 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-web" containerID="cri-o://b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649" gracePeriod=120 Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.673098 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy" containerID="cri-o://842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34" gracePeriod=120 Jul 31 18:46:38.673255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:38.673111 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="config-reloader" containerID="cri-o://cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd" gracePeriod=120 Jul 31 18:46:39.785015 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.784981 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917" exitCode=0 Jul 31 18:46:39.785015 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785012 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb" exitCode=0 Jul 31 18:46:39.785015 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785018 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34" exitCode=0 Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785026 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd" exitCode=0 Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785033 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157" exitCode=0 Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917"} Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785136 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb"} Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785161 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34"} Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785186 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd"} Jul 31 18:46:39.785363 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.785210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157"} Jul 31 18:46:39.916186 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:39.916157 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:40.100382 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100303 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100382 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100357 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100387 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100411 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100438 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100637 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100640 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100718 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100768 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22cwm\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100807 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.100854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100855 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.101095 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100892 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.101095 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.100916 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle\") pod \"12508d35-c009-41e8-865e-42fbaab02b2b\" (UID: \"12508d35-c009-41e8-865e-42fbaab02b2b\") " Jul 31 18:46:40.102048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.101791 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:40.102614 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.102563 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:46:40.102749 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.102719 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:40.104258 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.104233 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume" (OuterVolumeSpecName: "config-volume") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.104757 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.104549 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:46:40.104757 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.104608 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.104757 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.104692 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out" (OuterVolumeSpecName: "config-out") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:46:40.104925 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.104801 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.105322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.105292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm" (OuterVolumeSpecName: "kube-api-access-22cwm") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "kube-api-access-22cwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:46:40.105322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.105303 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.105480 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.105370 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.109717 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.109695 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.115352 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.115331 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config" (OuterVolumeSpecName: "web-config") pod "12508d35-c009-41e8-865e-42fbaab02b2b" (UID: "12508d35-c009-41e8-865e-42fbaab02b2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:40.201831 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201795 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.201831 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201822 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cwm\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-kube-api-access-22cwm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.201831 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201834 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201849 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-main-db\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201858 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-metrics-client-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201868 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12508d35-c009-41e8-865e-42fbaab02b2b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201876 2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-cluster-tls-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201887 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-web-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201894 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12508d35-c009-41e8-865e-42fbaab02b2b-config-out\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201902 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12508d35-c009-41e8-865e-42fbaab02b2b-tls-assets\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201910 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201924 2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.202046 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.201935 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/12508d35-c009-41e8-865e-42fbaab02b2b-secret-alertmanager-main-tls\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:40.792162 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.792130 2566 generic.go:356] "Generic (PLEG): container finished" podID="12508d35-c009-41e8-865e-42fbaab02b2b" containerID="b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649" exitCode=0 Jul 31 18:46:40.792466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.792201 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:40.792466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.792213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649"} Jul 31 18:46:40.792466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.792280 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"12508d35-c009-41e8-865e-42fbaab02b2b","Type":"ContainerDied","Data":"42bc4af96afcae2286c725539ecdc02213381bf4dfaed0fd87375d1324150418"} Jul 31 18:46:40.792466 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.792311 2566 scope.go:117] "RemoveContainer" containerID="3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917" Jul 31 18:46:40.800764 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.800719 2566 scope.go:117] "RemoveContainer" containerID="169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb" Jul 31 18:46:40.808158 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.808140 2566 scope.go:117] "RemoveContainer" containerID="842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34" Jul 31 18:46:40.815235 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.815219 2566 scope.go:117] "RemoveContainer" containerID="b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649" Jul 31 18:46:40.822158 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.822130 2566 scope.go:117] "RemoveContainer" containerID="cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd" Jul 31 18:46:40.825157 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.825137 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:40.829887 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.829867 2566 scope.go:117] "RemoveContainer" containerID="bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157" Jul 31 18:46:40.835724 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.835700 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:40.837669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.837652 2566 scope.go:117] "RemoveContainer" containerID="f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684" Jul 31 18:46:40.844813 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.844796 2566 scope.go:117] "RemoveContainer" containerID="3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917" Jul 31 18:46:40.845067 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.845048 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917\": container with ID starting with 3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917 not found: ID does not exist" containerID="3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917" Jul 31 18:46:40.845142 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845080 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917"} err="failed to get container status \"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917\": rpc error: code = NotFound desc = could not find container \"3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917\": container with ID starting with 3ddd1d5603faa8cbea25c8b24efba5d4636810ad17585e1e0396afc1e3fa1917 not found: ID does not exist" Jul 31 18:46:40.845142 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845108 2566 scope.go:117] "RemoveContainer" containerID="169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb" Jul 31 18:46:40.845345 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.845329 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb\": container with ID starting with 169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb not found: ID does not exist" containerID="169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb" Jul 31 18:46:40.845405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845352 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb"} err="failed to get container status \"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb\": rpc error: code = NotFound desc = could not find container \"169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb\": container with ID starting with 169d8d4434a6472cf54454e078bcad1e30727589f1c7580f276811a512631cbb not found: ID does not exist" Jul 31 18:46:40.845405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845372 2566 scope.go:117] "RemoveContainer" containerID="842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34" Jul 31 18:46:40.845711 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.845688 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34\": container with ID starting with 842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34 not found: ID does not exist" containerID="842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34" Jul 31 18:46:40.845782 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845716 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34"} err="failed to get container status \"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34\": rpc error: code = NotFound desc = could not find container \"842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34\": container with ID starting with 842d7a825cdc9a820b4f19ac5a02595458c4eb97d003d64714c84d0ea4d82a34 not found: ID does not exist" Jul 31 18:46:40.845782 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845735 2566 scope.go:117] "RemoveContainer" containerID="b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649" Jul 31 18:46:40.845955 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.845930 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649\": container with ID starting with b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649 not found: ID does not exist" containerID="b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649" Jul 31 18:46:40.846024 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845963 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649"} err="failed to get container status \"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649\": rpc error: code = NotFound desc = could not find container \"b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649\": container with ID starting with b9313d2f30b73e1c9967a46827e425a518b327363efeb9b46626c5b84e489649 not found: ID does not exist" Jul 31 18:46:40.846024 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.845982 2566 scope.go:117] "RemoveContainer" containerID="cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd" Jul 31 18:46:40.846227 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.846211 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd\": container with ID starting with cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd not found: ID does not exist" containerID="cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd" Jul 31 18:46:40.846275 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.846233 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd"} err="failed to get container status \"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd\": rpc error: code = NotFound desc = could not find container \"cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd\": container with ID starting with cef9fc978040c61fb56c1a55f17d32616965514c8a02c1c5fa596d24298095dd not found: ID does not exist" Jul 31 18:46:40.846275 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.846248 2566 scope.go:117] "RemoveContainer" containerID="bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157" Jul 31 18:46:40.846466 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.846447 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157\": container with ID starting with bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157 not found: ID does not exist" containerID="bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157" Jul 31 18:46:40.846533 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.846471 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157"} err="failed to get container status \"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157\": rpc error: code = NotFound desc = could not find container \"bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157\": container with ID starting with bfc858b4182d7d3652a4a75cd0ec1b8ea2ef9fc6d8ab4b76742c1f77d5d7f157 not found: ID does not exist" Jul 31 18:46:40.846533 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.846505 2566 scope.go:117] "RemoveContainer" containerID="f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684" Jul 31 18:46:40.846704 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:40.846690 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684\": container with ID starting with f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684 not found: ID does not exist" containerID="f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684" Jul 31 18:46:40.846747 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.846708 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684"} err="failed to get container status \"f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684\": rpc error: code = NotFound desc = could not find container \"f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684\": container with ID starting with f5d27a73fc10268e941b7b82280fa95bc0100a3739ea32d45478748f69e2f684 not found: ID does not exist" Jul 31 18:46:40.876990 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.876959 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:40.877270 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877258 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778f32d5-680d-42f9-8ee8-bfd5f073045f" containerName="console" Jul 31 18:46:40.877307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877271 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="778f32d5-680d-42f9-8ee8-bfd5f073045f" containerName="console" Jul 31 18:46:40.877307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877282 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="init-config-reloader" Jul 31 18:46:40.877307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877288 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="init-config-reloader" Jul 31 18:46:40.877307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877298 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy" Jul 31 18:46:40.877307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877304 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877312 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-web" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877317 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-web" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877323 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="prom-label-proxy" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877327 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="prom-label-proxy" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877334 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d425f4a-e7cd-492c-9913-bb8033384f79" containerName="console" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877338 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d425f4a-e7cd-492c-9913-bb8033384f79" containerName="console" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877347 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-metric" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877351 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-metric" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877359 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="alertmanager" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877365 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="alertmanager" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877371 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="config-reloader" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877376 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="config-reloader" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877383 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" containerName="console" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877388 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" containerName="console" Jul 31 18:46:40.877448 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877451 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="config-reloader" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877460 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="793c842e-a5bc-4ed9-995d-1a8c9c325b2c" containerName="console" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877466 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-metric" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877472 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d425f4a-e7cd-492c-9913-bb8033384f79" containerName="console" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877478 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="778f32d5-680d-42f9-8ee8-bfd5f073045f" containerName="console" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877483 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="alertmanager" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877505 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy-web" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877515 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="kube-rbac-proxy" Jul 31 18:46:40.877881 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.877522 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" containerName="prom-label-proxy" Jul 31 18:46:40.881166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.881150 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:40.884464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.884444 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 18:46:40.884464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.884458 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 18:46:40.884642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.884469 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 18:46:40.884642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.884458 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h76gt\"" Jul 31 18:46:40.884742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.884729 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 18:46:40.885031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.885008 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 18:46:40.885150 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.885012 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 18:46:40.885150 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.885130 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 18:46:40.885459 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.885442 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 18:46:40.893523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.891261 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 18:46:40.898055 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:40.898033 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:41.008459 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008417 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008470 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-web-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008629 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-out\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008669 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008875 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008901 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwljm\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-kube-api-access-vwljm\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.008938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.008927 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110031 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwljm\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-kube-api-access-vwljm\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110049 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-web-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-out\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.110690 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.110518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.111384 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.111014 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.111384 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.111026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.111384 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.111314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.113720 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.113402 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.113720 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.113536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.113720 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.113554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.114016 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.113957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.114016 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.113970 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.114016 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.114024 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-web-config\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.114016 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.114095 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-config-out\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.114436 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.114161 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.115556 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.115536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.119829 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.119811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwljm\" (UniqueName: \"kubernetes.io/projected/ee905a3d-0486-4d25-9af4-f79bf5c6d8b5-kube-api-access-vwljm\") pod \"alertmanager-main-0\" (UID: \"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.190835 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.190798 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 18:46:41.330642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.330606 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 18:46:41.332855 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:46:41.332831 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee905a3d_0486_4d25_9af4_f79bf5c6d8b5.slice/crio-319846006c9709e9c94dd744e2f5c79c5d3017b867acd29ff735a09a5fc54167 WatchSource:0}: Error finding container 319846006c9709e9c94dd744e2f5c79c5d3017b867acd29ff735a09a5fc54167: Status 404 returned error can't find the container with id 319846006c9709e9c94dd744e2f5c79c5d3017b867acd29ff735a09a5fc54167 Jul 31 18:46:41.798681 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.798649 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee905a3d-0486-4d25-9af4-f79bf5c6d8b5" containerID="8cb8cf719d5fbb4af497ce6b66db0374ec4fb1e59b9ff00dee164f13cf4af4d0" exitCode=0 Jul 31 18:46:41.799075 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.798687 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerDied","Data":"8cb8cf719d5fbb4af497ce6b66db0374ec4fb1e59b9ff00dee164f13cf4af4d0"} Jul 31 18:46:41.799075 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:41.798738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"319846006c9709e9c94dd744e2f5c79c5d3017b867acd29ff735a09a5fc54167"} Jul 31 18:46:42.035132 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.035104 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12508d35-c009-41e8-865e-42fbaab02b2b" path="/var/lib/kubelet/pods/12508d35-c009-41e8-865e-42fbaab02b2b/volumes" Jul 31 18:46:42.807955 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.807921 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"6dd8929cbbcf8feeea3ea9cc5ff57b5d729adeb9bf92eb10e7271d4245178b76"} Jul 31 18:46:42.808337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.808016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"eb62e48bb6d8514c0224147be302b7e8e12e6c2db68c1c82be5aeddc46081f1e"} Jul 31 18:46:42.808337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.808059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"83814262488cdecb6988ccfe5708cbe2c8902590549a77eda1ce86ca44d33f2b"} Jul 31 18:46:42.808337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.808089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"dcfe0592b007d54395f689b15c73479d085fc393fec023c0b705a91b09d9c71c"} Jul 31 18:46:42.808337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.808114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"4e75de6f6268fcb5c4b33f130b845be9036726747f17f353926203eecd5411f2"} Jul 31 18:46:42.808337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.808137 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee905a3d-0486-4d25-9af4-f79bf5c6d8b5","Type":"ContainerStarted","Data":"7fbe7fb7111abefd8d41b124ebc2a01dc117f911a6bb2b2a6fadef9d9f0433bb"} Jul 31 18:46:42.854647 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:42.854585 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.840012946 podStartE2EDuration="2.854563125s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.550179ms" isStatefulPod=false podCreationTimestamp="2026-07-31 18:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:46:42.851265791 +0000 UTC m=+151.389493442" watchObservedRunningTime="2026-07-31 18:46:42.854563125 +0000 UTC m=+151.392790777" Jul 31 18:46:43.054124 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.054090 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:43.054727 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.054675 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="prometheus" containerID="cri-o://67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" gracePeriod=600 Jul 31 18:46:43.055523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.054973 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy" containerID="cri-o://cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" gracePeriod=600 Jul 31 18:46:43.055523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.054996 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-web" containerID="cri-o://26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" gracePeriod=600 Jul 31 18:46:43.055523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.055014 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="thanos-sidecar" containerID="cri-o://10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" gracePeriod=600 Jul 31 18:46:43.055523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.055031 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="config-reloader" containerID="cri-o://3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" gracePeriod=600 Jul 31 18:46:43.055523 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.055116 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" gracePeriod=600 Jul 31 18:46:43.297866 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.297836 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.330823 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330727 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.330823 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330776 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.330823 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330806 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330858 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg6qq\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330887 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330930 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330955 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.330978 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331003 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331020 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331036 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331061 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331075 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331105 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331093 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331659 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331111 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331659 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331141 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.331957 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.331922 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.332184 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.332156 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.332946 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.332914 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.333110 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.333086 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.333247 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.333225 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.333360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.333338 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.333797 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.333765 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.334699 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.334670 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.335724 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.335699 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.335866 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.335761 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.336211 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.336176 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.336397 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.336367 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config" (OuterVolumeSpecName: "config") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.337290 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.337255 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq" (OuterVolumeSpecName: "kube-api-access-gg6qq") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "kube-api-access-gg6qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:46:43.337903 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.337872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.338345 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.338316 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.353756 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.351148 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config" (OuterVolumeSpecName: "web-config") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:46:43.432311 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.432311 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432314 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432333 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432350 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle\") pod \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\" (UID: \"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1\") " Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432475 2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432503 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432512 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432521 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432530 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432540 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432540 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-metrics-client-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432549 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432557 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432566 2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-grpc-tls\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432575 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432583 2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-kube-rbac-proxy\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432591 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-web-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432599 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6qq\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-kube-api-access-gg6qq\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432608 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432617 2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-secret-metrics-client-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432625 2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.432784 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.432724 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:46:43.433223 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.433205 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:46:43.434555 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.434532 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:46:43.434636 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.434553 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out" (OuterVolumeSpecName: "config-out") pod "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" (UID: "8bb2d8ce-c5a4-475d-9467-90aad9caa1d1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:46:43.533674 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.533637 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-tls-assets\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.533674 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.533667 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-prometheus-k8s-db\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.533674 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.533676 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-config-out\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.533892 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.533686 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814946 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" exitCode=0 Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814973 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" exitCode=0 Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814979 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" exitCode=0 Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814985 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" exitCode=0 Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814990 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" exitCode=0 Jul 31 18:46:43.814981 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.814995 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" exitCode=0 Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815038 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815188 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815243 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} Jul 31 18:46:43.815579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.815326 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bb2d8ce-c5a4-475d-9467-90aad9caa1d1","Type":"ContainerDied","Data":"aefc3558b526071ef2e24f489b9e80ffcf5bfda8109217944f2590db68a9ead8"} Jul 31 18:46:43.824007 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.823984 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.830957 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.830940 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.837893 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.837877 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.844554 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.844537 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.848364 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.848341 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:43.851517 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.851481 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.855286 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.855266 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:43.859028 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.859012 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.865840 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.865824 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.866103 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.866084 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.866172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866115 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.866172 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866141 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.866383 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.866367 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.866437 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866389 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.866437 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866409 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.866689 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.866671 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.866738 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866694 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.866738 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866709 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.866916 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.866897 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.866975 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866923 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.866975 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.866943 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.867155 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.867139 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.867189 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867160 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.867189 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867173 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.867378 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.867358 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.867415 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867382 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.867415 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867399 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.867655 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:46:43.867639 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.867714 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867658 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.867714 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867673 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.867882 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867865 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.867948 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.867884 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.868132 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868116 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.868181 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868133 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.868418 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868401 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.868478 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868419 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.868660 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868642 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.868698 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868660 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.868849 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868834 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.868890 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.868849 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.869059 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869045 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.869059 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869058 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.869282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869268 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.869282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869281 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.869559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869471 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.869559 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869505 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.869729 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869711 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.869775 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869731 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.869944 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869928 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.869986 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.869944 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.870166 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870143 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.870209 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870167 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.870387 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870370 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.870427 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870388 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.870635 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870619 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.870689 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870635 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.870858 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870844 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.870906 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.870858 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.871082 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871061 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.871119 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871086 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.871322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871304 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.871371 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871322 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.871531 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871516 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.871579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.871531 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.872062 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872044 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.872133 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872064 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.872332 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872314 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.872380 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872334 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.872572 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872556 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.872628 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872573 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.872767 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872745 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.872767 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872766 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.872972 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872956 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.873019 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.872971 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.873160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873146 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.873215 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873160 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.873364 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873348 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.873405 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873365 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.873586 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873567 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.873586 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873585 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.873786 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873771 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.873834 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873786 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.873985 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873971 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.874035 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.873984 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.874202 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874185 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.874258 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874203 2566 scope.go:117] "RemoveContainer" containerID="d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb" Jul 31 18:46:43.874440 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874422 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb"} err="failed to get container status \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": rpc error: code = NotFound desc = could not find container \"d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb\": container with ID starting with d71e8c6b6cee92a0e3fabc2886d1641ee489148dd25da881c1209085d9598dcb not found: ID does not exist" Jul 31 18:46:43.874506 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874440 2566 scope.go:117] "RemoveContainer" containerID="cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31" Jul 31 18:46:43.874684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874668 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31"} err="failed to get container status \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": rpc error: code = NotFound desc = could not find container \"cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31\": container with ID starting with cd3299f0b08c6725b1ac85b2d9f75a5d1aa9f5a67cbcae3e4f65be12a21e2f31 not found: ID does not exist" Jul 31 18:46:43.874684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874683 2566 scope.go:117] "RemoveContainer" containerID="26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50" Jul 31 18:46:43.874899 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874883 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50"} err="failed to get container status \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": rpc error: code = NotFound desc = could not find container \"26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50\": container with ID starting with 26900bd3a8598576ff0275a6c283f411c6f2f9e0b47899d5ca8d44ba143e5c50 not found: ID does not exist" Jul 31 18:46:43.874938 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.874900 2566 scope.go:117] "RemoveContainer" containerID="10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939" Jul 31 18:46:43.875118 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875101 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939"} err="failed to get container status \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": rpc error: code = NotFound desc = could not find container \"10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939\": container with ID starting with 10003512c4e6be04e499cfe3b7da11aff28c8c0bfa9414c41b44cc20c6303939 not found: ID does not exist" Jul 31 18:46:43.875118 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875118 2566 scope.go:117] "RemoveContainer" containerID="3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3" Jul 31 18:46:43.875330 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875310 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3"} err="failed to get container status \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": rpc error: code = NotFound desc = could not find container \"3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3\": container with ID starting with 3ace24d19bdefff9c1f75460c3121c2d7c1142c243c00deed0df1e17196b96a3 not found: ID does not exist" Jul 31 18:46:43.875401 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875332 2566 scope.go:117] "RemoveContainer" containerID="67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a" Jul 31 18:46:43.875566 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875549 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a"} err="failed to get container status \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": rpc error: code = NotFound desc = could not find container \"67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a\": container with ID starting with 67803c797a6fab1dd8c60a490ce2bc0eba7672c80f277c3066ca926991e33e3a not found: ID does not exist" Jul 31 18:46:43.875610 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875566 2566 scope.go:117] "RemoveContainer" containerID="1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041" Jul 31 18:46:43.875754 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.875737 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041"} err="failed to get container status \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": rpc error: code = NotFound desc = could not find container \"1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041\": container with ID starting with 1ca7e7eacdfbde02e8e5028d7ab5043481d17cde42e35dd382e774915ff23041 not found: ID does not exist" Jul 31 18:46:43.891642 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891617 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:43.891927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891916 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="init-config-reloader" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891929 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="init-config-reloader" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891936 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="prometheus" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891941 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="prometheus" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891949 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-web" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891954 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-web" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891964 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy" Jul 31 18:46:43.891968 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891970 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891983 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="thanos-sidecar" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891988 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="thanos-sidecar" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.891996 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="config-reloader" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892000 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="config-reloader" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892008 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-thanos" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892012 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-thanos" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892058 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892064 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-thanos" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892071 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="kube-rbac-proxy-web" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892079 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="config-reloader" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892085 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="prometheus" Jul 31 18:46:43.892160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.892090 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" containerName="thanos-sidecar" Jul 31 18:46:43.894380 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.894366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.899789 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.899712 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 31 18:46:43.899789 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.899736 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 31 18:46:43.899789 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.899780 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 31 18:46:43.900331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900135 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-npvzd\"" Jul 31 18:46:43.900331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900272 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 31 18:46:43.900331 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900314 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b88jmc869aguu\"" Jul 31 18:46:43.900544 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900436 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 31 18:46:43.900603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900589 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 31 18:46:43.900603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900597 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 31 18:46:43.900695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.900616 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 31 18:46:43.901286 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.901265 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 31 18:46:43.901724 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.901383 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 31 18:46:43.901724 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.901386 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 31 18:46:43.901724 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.901416 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 31 18:46:43.904660 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.904642 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 31 18:46:43.906972 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.906954 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 31 18:46:43.926613 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.923465 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:43.937498 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937506 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937730 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937797 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937811 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.937872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937851 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.938041 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9qg\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-kube-api-access-fz9qg\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.938041 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.938041 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:43.938041 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:43.937971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.034904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.034826 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb2d8ce-c5a4-475d-9467-90aad9caa1d1" path="/var/lib/kubelet/pods/8bb2d8ce-c5a4-475d-9467-90aad9caa1d1/volumes" Jul 31 18:46:44.039156 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039277 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039277 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039277 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039277 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9qg\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-kube-api-access-fz9qg\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039767 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039824 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.039962 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.039991 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.040011 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.040633 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.040181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.040633 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.040607 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.041973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.041093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.041973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.041661 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.042407 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.042378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.042558 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.042531 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.043687 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.043627 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.043687 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.043752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.043984 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.043958 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.044854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.044784 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.044854 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.044798 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.045236 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.045215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.045672 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.045653 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.045672 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.045662 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.045773 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.045734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-config\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.046322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.046300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.052034 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.052008 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9qg\" (UniqueName: \"kubernetes.io/projected/e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107-kube-api-access-fz9qg\") pod \"prometheus-k8s-0\" (UID: \"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.204979 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.204929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:46:44.350947 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.350781 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 31 18:46:44.353431 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:46:44.353403 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aaffbe_ca4c_4ab4_aa8f_92bf06c15107.slice/crio-4ff05c839117bc89b20751a0d401e8ce3d22a98cc3594dce33b25a92b0a5de31 WatchSource:0}: Error finding container 4ff05c839117bc89b20751a0d401e8ce3d22a98cc3594dce33b25a92b0a5de31: Status 404 returned error can't find the container with id 4ff05c839117bc89b20751a0d401e8ce3d22a98cc3594dce33b25a92b0a5de31 Jul 31 18:46:44.822014 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.821981 2566 generic.go:356] "Generic (PLEG): container finished" podID="e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107" containerID="ce1abfed6e611fa8ad1729ecb93a24fe7b9861172e718bc01dcc75bba13ad5de" exitCode=0 Jul 31 18:46:44.822422 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.822035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerDied","Data":"ce1abfed6e611fa8ad1729ecb93a24fe7b9861172e718bc01dcc75bba13ad5de"} Jul 31 18:46:44.822422 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:44.822101 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"4ff05c839117bc89b20751a0d401e8ce3d22a98cc3594dce33b25a92b0a5de31"} Jul 31 18:46:45.829884 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.829848 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"6090e32a1d4539917dc6d9a02bcc2b1b1842e5529b6dd5c3138e8fa41d78a7bf"} Jul 31 18:46:45.830282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.829939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"e2a7650c7902550f9604fd35a786d319b4abbdd0d0addb1009d3c5644923cb8c"} Jul 31 18:46:45.830282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.829966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"73af7fd9a25851262ba061ffa3a5e2ae4b4287679c78b0511e3053d23d9ba0f1"} Jul 31 18:46:45.830282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.829992 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"e4f1c788fa07c37630c699eaf5b7f26195c9f55edb9debdc2db3af6922342c96"} Jul 31 18:46:45.830282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.830015 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"670eb5f1d476c68c0e57b24ac887ac04369a4a9a86eb5e7968a87aa48fb160ba"} Jul 31 18:46:45.830282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.830039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107","Type":"ContainerStarted","Data":"9713bc593024426d73a39124ae940b7bceb078f1cd0d023b01471f0394880e67"} Jul 31 18:46:45.893338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:45.893280 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.874071563 podStartE2EDuration="2.893262854s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.191291ms" isStatefulPod=false podCreationTimestamp="2026-07-31 18:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:46:45.888964601 +0000 UTC m=+154.427192287" watchObservedRunningTime="2026-07-31 18:46:45.893262854 +0000 UTC m=+154.431490517" Jul 31 18:46:49.205543 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:46:49.205478 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:47:44.205680 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:47:44.205635 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:47:44.221716 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:47:44.221692 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:47:45.078727 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:47:45.078691 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 31 18:48:20.955524 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.955472 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld"] Jul 31 18:48:20.957773 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.957756 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:20.960657 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.960631 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 18:48:20.960738 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.960720 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xcv79\"" Jul 31 18:48:20.962133 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.962120 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 18:48:20.968617 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.968597 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld"] Jul 31 18:48:20.983570 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.983540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:20.983729 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:20.983620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bq5\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-kube-api-access-w8bq5\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.084799 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.084756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.084978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.084810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bq5\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-kube-api-access-w8bq5\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.099539 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.099506 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.099654 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.099583 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bq5\" (UniqueName: \"kubernetes.io/projected/28aef80b-679b-48d3-bd43-c99ce465d61c-kube-api-access-w8bq5\") pod \"cert-manager-cainjector-559cdd8c87-9v6ld\" (UID: \"28aef80b-679b-48d3-bd43-c99ce465d61c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.280612 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.280522 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" Jul 31 18:48:21.410948 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:21.410921 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld"] Jul 31 18:48:21.414071 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:48:21.414039 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28aef80b_679b_48d3_bd43_c99ce465d61c.slice/crio-c02bce4d8a987cc16e8354039163d5935aad3c1b00643e569071a15de4e88602 WatchSource:0}: Error finding container c02bce4d8a987cc16e8354039163d5935aad3c1b00643e569071a15de4e88602: Status 404 returned error can't find the container with id c02bce4d8a987cc16e8354039163d5935aad3c1b00643e569071a15de4e88602 Jul 31 18:48:22.208230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:22.208194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" event={"ID":"28aef80b-679b-48d3-bd43-c99ce465d61c","Type":"ContainerStarted","Data":"c02bce4d8a987cc16e8354039163d5935aad3c1b00643e569071a15de4e88602"} Jul 31 18:48:25.094548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.094464 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jvzk"] Jul 31 18:48:25.101456 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.100012 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.103752 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.103727 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zrdcs\"" Jul 31 18:48:25.108191 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.108169 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jvzk"] Jul 31 18:48:25.119830 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.119665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.119830 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.119702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzk6\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-kube-api-access-5nzk6\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.220322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.220287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.220322 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.220332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzk6\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-kube-api-access-5nzk6\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.225088 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.225054 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" event={"ID":"28aef80b-679b-48d3-bd43-c99ce465d61c","Type":"ContainerStarted","Data":"8d578b5b730e32e740916b7ffd8236acef062df5177e02986e93a265d43f5b98"} Jul 31 18:48:25.230723 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.230701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.230821 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.230766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzk6\" (UniqueName: \"kubernetes.io/projected/b62e51cd-4992-4848-8b65-33135954c27a-kube-api-access-5nzk6\") pod \"cert-manager-webhook-6486ffd997-6jvzk\" (UID: \"b62e51cd-4992-4848-8b65-33135954c27a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.246119 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.246076 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9v6ld" podStartSLOduration=1.9031888719999999 podStartE2EDuration="5.246060686s" totalImagesPullingTime="3.342871814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:48:25.24471759 +0000 UTC m=+253.782945244" watchObservedRunningTime="2026-07-31 18:48:25.246060686 +0000 UTC m=+253.784288318" Jul 31 18:48:25.410672 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.410645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:25.546539 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:25.546447 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jvzk"] Jul 31 18:48:25.548680 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:48:25.548657 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb62e51cd_4992_4848_8b65_33135954c27a.slice/crio-e4dc94e63e1cadf6b922d699c669722efdfc7c9696d799a8d76dd005c8698bad WatchSource:0}: Error finding container e4dc94e63e1cadf6b922d699c669722efdfc7c9696d799a8d76dd005c8698bad: Status 404 returned error can't find the container with id e4dc94e63e1cadf6b922d699c669722efdfc7c9696d799a8d76dd005c8698bad Jul 31 18:48:26.230621 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:26.230586 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" event={"ID":"b62e51cd-4992-4848-8b65-33135954c27a","Type":"ContainerStarted","Data":"626b90503a3da1798388ed9941364e8fd30a9769a8c85812a6784ee7712a9923"} Jul 31 18:48:26.231003 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:26.230630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" event={"ID":"b62e51cd-4992-4848-8b65-33135954c27a","Type":"ContainerStarted","Data":"e4dc94e63e1cadf6b922d699c669722efdfc7c9696d799a8d76dd005c8698bad"} Jul 31 18:48:26.252953 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:26.252906 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" podStartSLOduration=1.252889846 podStartE2EDuration="1.252889846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:48:26.252505389 +0000 UTC m=+254.790733031" watchObservedRunningTime="2026-07-31 18:48:26.252889846 +0000 UTC m=+254.791117495" Jul 31 18:48:27.234427 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:27.234398 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:33.240772 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:33.240694 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6jvzk" Jul 31 18:48:56.692304 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.692269 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9"] Jul 31 18:48:56.695866 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.695845 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.700129 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.700106 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 18:48:56.700316 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.700303 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6fdg\"" Jul 31 18:48:56.700546 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.700528 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 18:48:56.700635 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.700563 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 18:48:56.705810 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.705791 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 18:48:56.713468 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.713447 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9"] Jul 31 18:48:56.790421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.790384 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.790611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.790449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.790611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.790482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/288ea94a-9419-4a30-94ce-bafd8ccd8c63-kube-api-access-jmbqj\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.790611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.790541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.790611 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.790561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.891960 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.891908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.892149 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.891997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.892149 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.892035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/288ea94a-9419-4a30-94ce-bafd8ccd8c63-kube-api-access-jmbqj\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.892149 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.892085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.892149 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.892110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.892538 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.892477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.894437 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.894416 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/288ea94a-9419-4a30-94ce-bafd8ccd8c63-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.894716 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.894696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.894791 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.894722 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/288ea94a-9419-4a30-94ce-bafd8ccd8c63-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:56.905344 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:56.905315 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/288ea94a-9419-4a30-94ce-bafd8ccd8c63-kube-api-access-jmbqj\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-zk8c9\" (UID: \"288ea94a-9419-4a30-94ce-bafd8ccd8c63\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:57.006160 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:57.006058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:48:57.193034 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:57.190981 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9"] Jul 31 18:48:57.194244 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:48:57.194212 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288ea94a_9419_4a30_94ce_bafd8ccd8c63.slice/crio-2c8e497c771d39cc177a8a0a29e6e92157f1685e999e46eb9bffbec230ff5759 WatchSource:0}: Error finding container 2c8e497c771d39cc177a8a0a29e6e92157f1685e999e46eb9bffbec230ff5759: Status 404 returned error can't find the container with id 2c8e497c771d39cc177a8a0a29e6e92157f1685e999e46eb9bffbec230ff5759 Jul 31 18:48:57.364155 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:48:57.364077 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" event={"ID":"288ea94a-9419-4a30-94ce-bafd8ccd8c63","Type":"ContainerStarted","Data":"2c8e497c771d39cc177a8a0a29e6e92157f1685e999e46eb9bffbec230ff5759"} Jul 31 18:49:00.381075 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:00.381034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" event={"ID":"288ea94a-9419-4a30-94ce-bafd8ccd8c63","Type":"ContainerStarted","Data":"0f15618f3b5fd2dc283e5940e6f9bd93f0e4825ffc92321997cf3d8cf813151b"} Jul 31 18:49:02.391368 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:02.391334 2566 generic.go:356] "Generic (PLEG): container finished" podID="288ea94a-9419-4a30-94ce-bafd8ccd8c63" containerID="0f15618f3b5fd2dc283e5940e6f9bd93f0e4825ffc92321997cf3d8cf813151b" exitCode=0 Jul 31 18:49:02.391783 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:02.391414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" event={"ID":"288ea94a-9419-4a30-94ce-bafd8ccd8c63","Type":"ContainerDied","Data":"0f15618f3b5fd2dc283e5940e6f9bd93f0e4825ffc92321997cf3d8cf813151b"} Jul 31 18:49:03.398117 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:03.398086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" event={"ID":"288ea94a-9419-4a30-94ce-bafd8ccd8c63","Type":"ContainerStarted","Data":"c54030918672ea2a499849cc350247270ce699a59b6794eada21e1501639d40c"} Jul 31 18:49:03.398525 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:03.398298 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:49:03.425199 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:03.425151 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" podStartSLOduration=2.603625966 podStartE2EDuration="7.425137387s" totalImagesPullingTime="3.780232326s" totalInitContainerRuntime="1.041279095s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:03.422422007 +0000 UTC m=+291.960649668" watchObservedRunningTime="2026-07-31 18:49:03.425137387 +0000 UTC m=+291.963365037" Jul 31 18:49:11.883643 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:11.883618 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 18:49:14.405548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:14.405517 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-zk8c9" Jul 31 18:49:15.324709 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.324675 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26"] Jul 31 18:49:15.327717 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.327701 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.333514 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333470 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 31 18:49:15.333514 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333483 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 31 18:49:15.333685 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333470 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 18:49:15.333685 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333473 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 31 18:49:15.333685 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333475 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dp8kl\"" Jul 31 18:49:15.333685 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.333602 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 18:49:15.344893 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.344867 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26"] Jul 31 18:49:15.362993 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.362963 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.363133 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.363031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a209c091-7485-4aaf-a01d-cd80d74952c6-manager-config\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.363133 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.363078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfptz\" (UniqueName: \"kubernetes.io/projected/a209c091-7485-4aaf-a01d-cd80d74952c6-kube-api-access-bfptz\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.363222 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.363142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.463905 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.463871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfptz\" (UniqueName: \"kubernetes.io/projected/a209c091-7485-4aaf-a01d-cd80d74952c6-kube-api-access-bfptz\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.464374 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.463910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.464374 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.463965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.464374 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.464039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a209c091-7485-4aaf-a01d-cd80d74952c6-manager-config\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.464789 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.464768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a209c091-7485-4aaf-a01d-cd80d74952c6-manager-config\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.466547 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.466528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.466759 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.466737 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a209c091-7485-4aaf-a01d-cd80d74952c6-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.474449 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.474424 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfptz\" (UniqueName: \"kubernetes.io/projected/a209c091-7485-4aaf-a01d-cd80d74952c6-kube-api-access-bfptz\") pod \"lws-controller-manager-6478f7cf66-kht26\" (UID: \"a209c091-7485-4aaf-a01d-cd80d74952c6\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.637616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.637586 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:15.784851 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.784816 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26"] Jul 31 18:49:15.789341 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:49:15.789319 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda209c091_7485_4aaf_a01d_cd80d74952c6.slice/crio-f2d6ba599f8cfb1e5db0f8330b8d1c6e04e0ba8640e72199830861cd7dd1534b WatchSource:0}: Error finding container f2d6ba599f8cfb1e5db0f8330b8d1c6e04e0ba8640e72199830861cd7dd1534b: Status 404 returned error can't find the container with id f2d6ba599f8cfb1e5db0f8330b8d1c6e04e0ba8640e72199830861cd7dd1534b Jul 31 18:49:15.791254 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:15.791238 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:49:16.455304 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.455273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" event={"ID":"a209c091-7485-4aaf-a01d-cd80d74952c6","Type":"ContainerStarted","Data":"f2d6ba599f8cfb1e5db0f8330b8d1c6e04e0ba8640e72199830861cd7dd1534b"} Jul 31 18:49:16.818186 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.817965 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-phcjk"] Jul 31 18:49:16.821795 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.821771 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.828121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.828099 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 18:49:16.828978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.828633 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gqjql\"" Jul 31 18:49:16.828978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.828633 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 18:49:16.828978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.828838 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 18:49:16.830576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.830387 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 18:49:16.854715 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.854688 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-phcjk"] Jul 31 18:49:16.876132 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.875900 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.876132 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.875943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xvb\" (UniqueName: \"kubernetes.io/projected/85dcf311-dd28-4e35-ab25-302504bc80b7-kube-api-access-75xvb\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.876132 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.876010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85dcf311-dd28-4e35-ab25-302504bc80b7-tmp\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.977128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.977090 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.977326 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.977142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xvb\" (UniqueName: \"kubernetes.io/projected/85dcf311-dd28-4e35-ab25-302504bc80b7-kube-api-access-75xvb\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.977326 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.977194 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85dcf311-dd28-4e35-ab25-302504bc80b7-tmp\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.977326 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:49:16.977317 2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 31 18:49:16.977524 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:49:16.977392 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs podName:85dcf311-dd28-4e35-ab25-302504bc80b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:49:17.477368598 +0000 UTC m=+306.015596243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs") pod "kube-auth-proxy-86dbc777-phcjk" (UID: "85dcf311-dd28-4e35-ab25-302504bc80b7") : secret "kube-auth-proxy-tls" not found Jul 31 18:49:16.980094 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.980044 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85dcf311-dd28-4e35-ab25-302504bc80b7-tmp\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:16.991579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:16.991551 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xvb\" (UniqueName: \"kubernetes.io/projected/85dcf311-dd28-4e35-ab25-302504bc80b7-kube-api-access-75xvb\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:17.481307 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:17.481266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:17.484364 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:17.484338 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dcf311-dd28-4e35-ab25-302504bc80b7-tls-certs\") pod \"kube-auth-proxy-86dbc777-phcjk\" (UID: \"85dcf311-dd28-4e35-ab25-302504bc80b7\") " pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:17.736337 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:17.736264 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" Jul 31 18:49:18.404271 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:18.404244 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dbc777-phcjk"] Jul 31 18:49:18.407006 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:49:18.406984 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dcf311_dd28_4e35_ab25_302504bc80b7.slice/crio-0ecd51e7275419dfac3655dde9f80ec1e8e5bf5e2adccb5bb40332491d26b958 WatchSource:0}: Error finding container 0ecd51e7275419dfac3655dde9f80ec1e8e5bf5e2adccb5bb40332491d26b958: Status 404 returned error can't find the container with id 0ecd51e7275419dfac3655dde9f80ec1e8e5bf5e2adccb5bb40332491d26b958 Jul 31 18:49:18.480123 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:18.480078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" event={"ID":"85dcf311-dd28-4e35-ab25-302504bc80b7","Type":"ContainerStarted","Data":"0ecd51e7275419dfac3655dde9f80ec1e8e5bf5e2adccb5bb40332491d26b958"} Jul 31 18:49:18.482617 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:18.482587 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" event={"ID":"a209c091-7485-4aaf-a01d-cd80d74952c6","Type":"ContainerStarted","Data":"0aa8540051472eeb4c6a8402b82294f89ebfb761084d7a8b574613d87dbabb4a"} Jul 31 18:49:18.482766 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:18.482657 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:49:18.501351 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:18.501301 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" podStartSLOduration=0.949800601 podStartE2EDuration="3.501285958s" totalImagesPullingTime="2.551485357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:18.500536107 +0000 UTC m=+307.038763754" watchObservedRunningTime="2026-07-31 18:49:18.501285958 +0000 UTC m=+307.039513608" Jul 31 18:49:21.499636 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:21.499598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" event={"ID":"85dcf311-dd28-4e35-ab25-302504bc80b7","Type":"ContainerStarted","Data":"2023b86f7bf78df5a666db44dd3ceb5735da27f0d87b0dec6ee49793304c8899"} Jul 31 18:49:21.522073 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:21.522019 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dbc777-phcjk" podStartSLOduration=3.341867652 podStartE2EDuration="5.522004369s" totalImagesPullingTime="2.180136717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:21.519882279 +0000 UTC m=+310.058109941" watchObservedRunningTime="2026-07-31 18:49:21.522004369 +0000 UTC m=+310.060232020" Jul 31 18:49:29.491388 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:49:29.491355 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-kht26" Jul 31 18:50:43.620823 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.620743 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb"] Jul 31 18:50:43.624013 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.623999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.628269 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.628245 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 18:50:43.628377 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.628252 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5kvcw\"" Jul 31 18:50:43.628586 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.628570 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 18:50:43.641904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.641881 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb"] Jul 31 18:50:43.657225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.657192 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.657333 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.657235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhnt\" (UniqueName: \"kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.758444 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.758412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.758639 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.758451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhnt\" (UniqueName: \"kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.758820 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.758799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.777303 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.777267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhnt\" (UniqueName: \"kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt\") pod \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:43.934600 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:43.934558 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:44.090012 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:44.089990 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb"] Jul 31 18:50:44.092174 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:50:44.092142 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9776010_c76b_4a5f_9294_f0a08a649ecd.slice/crio-7e37700b4735600965b8f04ecf1ddf5924f333d9da9a95473f90d93c9bd734df WatchSource:0}: Error finding container 7e37700b4735600965b8f04ecf1ddf5924f333d9da9a95473f90d93c9bd734df: Status 404 returned error can't find the container with id 7e37700b4735600965b8f04ecf1ddf5924f333d9da9a95473f90d93c9bd734df Jul 31 18:50:44.879315 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:44.879278 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" event={"ID":"b9776010-c76b-4a5f-9294-f0a08a649ecd","Type":"ContainerStarted","Data":"7e37700b4735600965b8f04ecf1ddf5924f333d9da9a95473f90d93c9bd734df"} Jul 31 18:50:48.901695 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:48.901657 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" event={"ID":"b9776010-c76b-4a5f-9294-f0a08a649ecd","Type":"ContainerStarted","Data":"b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e"} Jul 31 18:50:48.902068 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:48.901774 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:50:48.958867 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:48.958762 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" podStartSLOduration=1.364803826 podStartE2EDuration="5.958745428s" totalImagesPullingTime="4.593941602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:50:48.957146361 +0000 UTC m=+397.495374011" watchObservedRunningTime="2026-07-31 18:50:48.958745428 +0000 UTC m=+397.496973079" Jul 31 18:50:59.908969 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:50:59.908936 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.487970 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.487938 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb"] Jul 31 18:51:01.488409 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.488139 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" containerName="manager" containerID="cri-o://b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e" gracePeriod=2 Jul 31 18:51:01.493871 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.493838 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb"] Jul 31 18:51:01.518203 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.518174 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:01.518779 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.518758 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" containerName="manager" Jul 31 18:51:01.518779 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.518783 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" containerName="manager" Jul 31 18:51:01.518930 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.518897 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" containerName="manager" Jul 31 18:51:01.522171 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.522151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.526662 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:01.526626 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.541975 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.541952 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:01.618925 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.618887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szmx\" (UniqueName: \"kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.619076 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.618941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.719912 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.719886 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szmx\" (UniqueName: \"kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.720025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.719925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.720309 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.720283 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.734324 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.734290 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.735756 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.735734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szmx\" (UniqueName: \"kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx\") pod \"kuadrant-operator-controller-manager-58fc56757c-7n7h8\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.737283 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:01.737261 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.820872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.820765 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkhnt\" (UniqueName: \"kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt\") pod \"b9776010-c76b-4a5f-9294-f0a08a649ecd\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " Jul 31 18:51:01.820872 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.820819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume\") pod \"b9776010-c76b-4a5f-9294-f0a08a649ecd\" (UID: \"b9776010-c76b-4a5f-9294-f0a08a649ecd\") " Jul 31 18:51:01.821306 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.821281 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9776010-c76b-4a5f-9294-f0a08a649ecd" (UID: "b9776010-c76b-4a5f-9294-f0a08a649ecd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:51:01.823180 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.823157 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt" (OuterVolumeSpecName: "kube-api-access-hkhnt") pod "b9776010-c76b-4a5f-9294-f0a08a649ecd" (UID: "b9776010-c76b-4a5f-9294-f0a08a649ecd"). InnerVolumeSpecName "kube-api-access-hkhnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:01.862555 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.862511 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:01.922434 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.922393 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhnt\" (UniqueName: \"kubernetes.io/projected/b9776010-c76b-4a5f-9294-f0a08a649ecd-kube-api-access-hkhnt\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:51:01.922434 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.922431 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9776010-c76b-4a5f-9294-f0a08a649ecd-extensions-socket-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:51:01.972878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.972849 2566 generic.go:356] "Generic (PLEG): container finished" podID="b9776010-c76b-4a5f-9294-f0a08a649ecd" containerID="b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e" exitCode=0 Jul 31 18:51:01.973051 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.972914 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.973051 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.972960 2566 scope.go:117] "RemoveContainer" containerID="b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e" Jul 31 18:51:01.975923 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:01.975887 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:01.982325 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.982300 2566 scope.go:117] "RemoveContainer" containerID="b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e" Jul 31 18:51:01.983037 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:01.983009 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e\": container with ID starting with b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e not found: ID does not exist" containerID="b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e" Jul 31 18:51:01.983213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:01.983165 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e"} err="failed to get container status \"b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e\": rpc error: code = NotFound desc = could not find container \"b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e\": container with ID starting with b0eb1ecc7ad87ff91020696eb0635f68ebe41f47839e2a297e8883c54a5c416e not found: ID does not exist" Jul 31 18:51:01.985996 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:01.985968 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:02.008421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:02.008385 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:02.010941 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:51:02.010915 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afc456f_3551_4b53_aee0_6afb8c41baed.slice/crio-50777b2900dd252848e7b7c1a9e121c2987ef4e04060edfc182ab6ac3f654b03 WatchSource:0}: Error finding container 50777b2900dd252848e7b7c1a9e121c2987ef4e04060edfc182ab6ac3f654b03: Status 404 returned error can't find the container with id 50777b2900dd252848e7b7c1a9e121c2987ef4e04060edfc182ab6ac3f654b03 Jul 31 18:51:02.034921 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:02.034882 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-drdlb\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-drdlb" Jul 31 18:51:02.036987 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:02.036959 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9776010-c76b-4a5f-9294-f0a08a649ecd" path="/var/lib/kubelet/pods/b9776010-c76b-4a5f-9294-f0a08a649ecd/volumes" Jul 31 18:51:02.979574 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:02.979537 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" event={"ID":"5afc456f-3551-4b53-aee0-6afb8c41baed","Type":"ContainerStarted","Data":"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67"} Jul 31 18:51:02.980004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:02.979631 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" event={"ID":"5afc456f-3551-4b53-aee0-6afb8c41baed","Type":"ContainerStarted","Data":"50777b2900dd252848e7b7c1a9e121c2987ef4e04060edfc182ab6ac3f654b03"} Jul 31 18:51:02.980004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:02.979662 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:03.007614 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:03.007554 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" podStartSLOduration=2.007534922 podStartE2EDuration="2.007534922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:03.005136023 +0000 UTC m=+411.543363673" watchObservedRunningTime="2026-07-31 18:51:03.007534922 +0000 UTC m=+411.545762572" Jul 31 18:51:13.987820 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:13.987783 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:16.954878 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:16.954843 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:16.955315 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:16.955060 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" podUID="5afc456f-3551-4b53-aee0-6afb8c41baed" containerName="manager" containerID="cri-o://13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67" gracePeriod=10 Jul 31 18:51:17.207985 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.207925 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:17.361583 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.361550 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2szmx\" (UniqueName: \"kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx\") pod \"5afc456f-3551-4b53-aee0-6afb8c41baed\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " Jul 31 18:51:17.361745 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.361662 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume\") pod \"5afc456f-3551-4b53-aee0-6afb8c41baed\" (UID: \"5afc456f-3551-4b53-aee0-6afb8c41baed\") " Jul 31 18:51:17.362019 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.361989 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5afc456f-3551-4b53-aee0-6afb8c41baed" (UID: "5afc456f-3551-4b53-aee0-6afb8c41baed"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 18:51:17.363806 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.363781 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx" (OuterVolumeSpecName: "kube-api-access-2szmx") pod "5afc456f-3551-4b53-aee0-6afb8c41baed" (UID: "5afc456f-3551-4b53-aee0-6afb8c41baed"). InnerVolumeSpecName "kube-api-access-2szmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:17.463192 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.463107 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5afc456f-3551-4b53-aee0-6afb8c41baed-extensions-socket-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:51:17.463192 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:17.463143 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szmx\" (UniqueName: \"kubernetes.io/projected/5afc456f-3551-4b53-aee0-6afb8c41baed-kube-api-access-2szmx\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:51:18.052255 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.052221 2566 generic.go:356] "Generic (PLEG): container finished" podID="5afc456f-3551-4b53-aee0-6afb8c41baed" containerID="13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67" exitCode=0 Jul 31 18:51:18.052700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.052267 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" Jul 31 18:51:18.052700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.052306 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" event={"ID":"5afc456f-3551-4b53-aee0-6afb8c41baed","Type":"ContainerDied","Data":"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67"} Jul 31 18:51:18.052700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.052355 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8" event={"ID":"5afc456f-3551-4b53-aee0-6afb8c41baed","Type":"ContainerDied","Data":"50777b2900dd252848e7b7c1a9e121c2987ef4e04060edfc182ab6ac3f654b03"} Jul 31 18:51:18.052700 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.052382 2566 scope.go:117] "RemoveContainer" containerID="13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67" Jul 31 18:51:18.060884 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.060865 2566 scope.go:117] "RemoveContainer" containerID="13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67" Jul 31 18:51:18.061162 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:18.061141 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67\": container with ID starting with 13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67 not found: ID does not exist" containerID="13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67" Jul 31 18:51:18.061214 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.061180 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67"} err="failed to get container status \"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67\": rpc error: code = NotFound desc = could not find container \"13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67\": container with ID starting with 13cf03fbfd776803c1bd6de0c6de552b1b3e0bbbe935b3475920a146d199ce67 not found: ID does not exist" Jul 31 18:51:18.083199 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.083169 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:18.087616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:18.087589 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7n7h8"] Jul 31 18:51:20.034260 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:20.034224 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afc456f-3551-4b53-aee0-6afb8c41baed" path="/var/lib/kubelet/pods/5afc456f-3551-4b53-aee0-6afb8c41baed/volumes" Jul 31 18:51:51.341639 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.341605 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:51:51.342095 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.342050 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afc456f-3551-4b53-aee0-6afb8c41baed" containerName="manager" Jul 31 18:51:51.342095 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.342063 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afc456f-3551-4b53-aee0-6afb8c41baed" containerName="manager" Jul 31 18:51:51.342209 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.342127 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afc456f-3551-4b53-aee0-6afb8c41baed" containerName="manager" Jul 31 18:51:51.344224 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.344210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.347159 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.347139 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 18:51:51.347159 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.347152 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 18:51:51.348612 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.348594 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-57xp7\"" Jul 31 18:51:51.348710 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.348614 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 18:51:51.354419 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.354397 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:51:51.436912 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.436878 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:51:51.463449 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.463418 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvgx\" (UniqueName: \"kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.463623 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.463561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.564380 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.564304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.564380 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.564362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvgx\" (UniqueName: \"kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.564903 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.564882 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.574435 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.574410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvgx\" (UniqueName: \"kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx\") pod \"limitador-limitador-859c4d96d6-q8bvm\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.655048 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.655010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:51.792237 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:51.792215 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:51:51.794286 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:51:51.794257 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144a3552_df00_45d4_aad0_ee3c89023a88.slice/crio-b300ed42751ae787ed94ef8db0f191967c0fb017c37d59ac248bdcdbaf37ccb1 WatchSource:0}: Error finding container b300ed42751ae787ed94ef8db0f191967c0fb017c37d59ac248bdcdbaf37ccb1: Status 404 returned error can't find the container with id b300ed42751ae787ed94ef8db0f191967c0fb017c37d59ac248bdcdbaf37ccb1 Jul 31 18:51:52.205744 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.205711 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:52.208765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.208745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:52.211531 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.211509 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s7w92\"" Jul 31 18:51:52.217114 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.217066 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:52.219978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.219950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" event={"ID":"144a3552-df00-45d4-aad0-ee3c89023a88","Type":"ContainerStarted","Data":"b300ed42751ae787ed94ef8db0f191967c0fb017c37d59ac248bdcdbaf37ccb1"} Jul 31 18:51:52.270025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.269996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsvw\" (UniqueName: \"kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw\") pod \"authorino-b54df94b4-6xt6f\" (UID: \"19f137c7-1e51-4451-b18e-991c1c511ec4\") " pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:52.371053 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.370952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsvw\" (UniqueName: \"kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw\") pod \"authorino-b54df94b4-6xt6f\" (UID: \"19f137c7-1e51-4451-b18e-991c1c511ec4\") " pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:52.381420 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.381394 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsvw\" (UniqueName: \"kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw\") pod \"authorino-b54df94b4-6xt6f\" (UID: \"19f137c7-1e51-4451-b18e-991c1c511ec4\") " pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:52.519957 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.519878 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:52.651481 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:52.649929 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:53.226684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:53.226644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-6xt6f" event={"ID":"19f137c7-1e51-4451-b18e-991c1c511ec4","Type":"ContainerStarted","Data":"99ed6adb59ae10e9514d359c2315a53f1717cb0078d3adbbd5e4ec7ed8343664"} Jul 31 18:51:55.147736 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.147698 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:55.237621 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.237582 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-6xt6f" event={"ID":"19f137c7-1e51-4451-b18e-991c1c511ec4","Type":"ContainerStarted","Data":"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21"} Jul 31 18:51:55.239127 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.239103 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" event={"ID":"144a3552-df00-45d4-aad0-ee3c89023a88","Type":"ContainerStarted","Data":"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc"} Jul 31 18:51:55.239240 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.239156 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:51:55.255521 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.255447 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-6xt6f" podStartSLOduration=1.253814529 podStartE2EDuration="3.255434182s" totalImagesPullingTime="2.001619653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:55.253186223 +0000 UTC m=+463.791413884" watchObservedRunningTime="2026-07-31 18:51:55.255434182 +0000 UTC m=+463.793661833" Jul 31 18:51:55.272541 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:55.272471 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" podStartSLOduration=1.418660918 podStartE2EDuration="4.272457361s" totalImagesPullingTime="2.853796443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:55.272199714 +0000 UTC m=+463.810427366" watchObservedRunningTime="2026-07-31 18:51:55.272457361 +0000 UTC m=+463.810685011" Jul 31 18:51:56.243481 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:56.243429 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-6xt6f" podUID="19f137c7-1e51-4451-b18e-991c1c511ec4" containerName="authorino" containerID="cri-o://ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21" gracePeriod=30 Jul 31 18:51:56.487562 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:56.487540 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:56.614280 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:56.614193 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqsvw\" (UniqueName: \"kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw\") pod \"19f137c7-1e51-4451-b18e-991c1c511ec4\" (UID: \"19f137c7-1e51-4451-b18e-991c1c511ec4\") " Jul 31 18:51:56.616333 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:56.616301 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw" (OuterVolumeSpecName: "kube-api-access-fqsvw") pod "19f137c7-1e51-4451-b18e-991c1c511ec4" (UID: "19f137c7-1e51-4451-b18e-991c1c511ec4"). InnerVolumeSpecName "kube-api-access-fqsvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:51:56.715797 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:56.715762 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqsvw\" (UniqueName: \"kubernetes.io/projected/19f137c7-1e51-4451-b18e-991c1c511ec4-kube-api-access-fqsvw\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:51:57.248713 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.248674 2566 generic.go:356] "Generic (PLEG): container finished" podID="19f137c7-1e51-4451-b18e-991c1c511ec4" containerID="ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21" exitCode=0 Jul 31 18:51:57.249128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.248720 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-6xt6f" Jul 31 18:51:57.249128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.248765 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-6xt6f" event={"ID":"19f137c7-1e51-4451-b18e-991c1c511ec4","Type":"ContainerDied","Data":"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21"} Jul 31 18:51:57.249128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.248805 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-6xt6f" event={"ID":"19f137c7-1e51-4451-b18e-991c1c511ec4","Type":"ContainerDied","Data":"99ed6adb59ae10e9514d359c2315a53f1717cb0078d3adbbd5e4ec7ed8343664"} Jul 31 18:51:57.249128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.248826 2566 scope.go:117] "RemoveContainer" containerID="ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21" Jul 31 18:51:57.257614 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.257597 2566 scope.go:117] "RemoveContainer" containerID="ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21" Jul 31 18:51:57.257845 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:51:57.257828 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21\": container with ID starting with ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21 not found: ID does not exist" containerID="ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21" Jul 31 18:51:57.257886 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.257853 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21"} err="failed to get container status \"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21\": rpc error: code = NotFound desc = could not find container \"ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21\": container with ID starting with ee0d4c5596754684cf4c5ec09c6a1d3a8ac6b03d4034eb9b629561d47853ff21 not found: ID does not exist" Jul 31 18:51:57.271776 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.271752 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:57.276552 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:57.276533 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-6xt6f"] Jul 31 18:51:58.034616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:51:58.034583 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f137c7-1e51-4451-b18e-991c1c511ec4" path="/var/lib/kubelet/pods/19f137c7-1e51-4451-b18e-991c1c511ec4/volumes" Jul 31 18:52:05.148098 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.148056 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:52:05.148630 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.148311 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" podUID="144a3552-df00-45d4-aad0-ee3c89023a88" containerName="limitador" containerID="cri-o://6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc" gracePeriod=30 Jul 31 18:52:05.149061 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.148961 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:52:05.696684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.696662 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:52:05.799868 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.799771 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbvgx\" (UniqueName: \"kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx\") pod \"144a3552-df00-45d4-aad0-ee3c89023a88\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " Jul 31 18:52:05.799868 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.799850 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file\") pod \"144a3552-df00-45d4-aad0-ee3c89023a88\" (UID: \"144a3552-df00-45d4-aad0-ee3c89023a88\") " Jul 31 18:52:05.800213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.800187 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file" (OuterVolumeSpecName: "config-file") pod "144a3552-df00-45d4-aad0-ee3c89023a88" (UID: "144a3552-df00-45d4-aad0-ee3c89023a88"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 18:52:05.802074 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.802050 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx" (OuterVolumeSpecName: "kube-api-access-tbvgx") pod "144a3552-df00-45d4-aad0-ee3c89023a88" (UID: "144a3552-df00-45d4-aad0-ee3c89023a88"). InnerVolumeSpecName "kube-api-access-tbvgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:52:05.900930 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.900895 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbvgx\" (UniqueName: \"kubernetes.io/projected/144a3552-df00-45d4-aad0-ee3c89023a88-kube-api-access-tbvgx\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:05.900930 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:05.900923 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/144a3552-df00-45d4-aad0-ee3c89023a88-config-file\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:06.293168 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.293135 2566 generic.go:356] "Generic (PLEG): container finished" podID="144a3552-df00-45d4-aad0-ee3c89023a88" containerID="6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc" exitCode=0 Jul 31 18:52:06.293576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.293178 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" Jul 31 18:52:06.293576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.293203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" event={"ID":"144a3552-df00-45d4-aad0-ee3c89023a88","Type":"ContainerDied","Data":"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc"} Jul 31 18:52:06.293576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.293242 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q8bvm" event={"ID":"144a3552-df00-45d4-aad0-ee3c89023a88","Type":"ContainerDied","Data":"b300ed42751ae787ed94ef8db0f191967c0fb017c37d59ac248bdcdbaf37ccb1"} Jul 31 18:52:06.293576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.293264 2566 scope.go:117] "RemoveContainer" containerID="6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc" Jul 31 18:52:06.301525 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.301503 2566 scope.go:117] "RemoveContainer" containerID="6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc" Jul 31 18:52:06.301804 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:52:06.301782 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc\": container with ID starting with 6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc not found: ID does not exist" containerID="6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc" Jul 31 18:52:06.301896 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.301811 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc"} err="failed to get container status \"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc\": rpc error: code = NotFound desc = could not find container \"6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc\": container with ID starting with 6850d432824143d91ceee3b439e082bef690b12580781e048019b2ecd11076bc not found: ID does not exist" Jul 31 18:52:06.325273 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.325244 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:52:06.330570 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.330550 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q8bvm"] Jul 31 18:52:06.893963 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.893934 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xtszm"] Jul 31 18:52:06.894300 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894287 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144a3552-df00-45d4-aad0-ee3c89023a88" containerName="limitador" Jul 31 18:52:06.894360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894302 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="144a3552-df00-45d4-aad0-ee3c89023a88" containerName="limitador" Jul 31 18:52:06.894360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894321 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f137c7-1e51-4451-b18e-991c1c511ec4" containerName="authorino" Jul 31 18:52:06.894360 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894328 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f137c7-1e51-4451-b18e-991c1c511ec4" containerName="authorino" Jul 31 18:52:06.894461 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894387 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="144a3552-df00-45d4-aad0-ee3c89023a88" containerName="limitador" Jul 31 18:52:06.894461 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.894394 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f137c7-1e51-4451-b18e-991c1c511ec4" containerName="authorino" Jul 31 18:52:06.898569 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.898545 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:06.903484 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.903461 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2vhnt\"" Jul 31 18:52:06.903653 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.903516 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 31 18:52:06.903701 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.903661 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 18:52:06.904760 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.904737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 31 18:52:06.914256 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:06.914235 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xtszm"] Jul 31 18:52:07.010218 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.010180 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfssn\" (UniqueName: \"kubernetes.io/projected/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-kube-api-access-rfssn\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.010409 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.010224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-data\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.111362 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.111320 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfssn\" (UniqueName: \"kubernetes.io/projected/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-kube-api-access-rfssn\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.111362 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.111363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-data\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.111781 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.111764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-data\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.121745 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.121721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfssn\" (UniqueName: \"kubernetes.io/projected/8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935-kube-api-access-rfssn\") pod \"postgres-69fb4594cf-xtszm\" (UID: \"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.209744 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.209655 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:07.343443 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:07.343420 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xtszm"] Jul 31 18:52:07.345452 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:07.345426 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4b6a1a_a2a1_48cd_bab3_eb7d72d6d935.slice/crio-388913317ddc0c3917f4db5429e39f8792ab940e44c2d298dac92327c47d62e3 WatchSource:0}: Error finding container 388913317ddc0c3917f4db5429e39f8792ab940e44c2d298dac92327c47d62e3: Status 404 returned error can't find the container with id 388913317ddc0c3917f4db5429e39f8792ab940e44c2d298dac92327c47d62e3 Jul 31 18:52:08.038643 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:08.036638 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144a3552-df00-45d4-aad0-ee3c89023a88" path="/var/lib/kubelet/pods/144a3552-df00-45d4-aad0-ee3c89023a88/volumes" Jul 31 18:52:08.306616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:08.306471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" event={"ID":"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935","Type":"ContainerStarted","Data":"388913317ddc0c3917f4db5429e39f8792ab940e44c2d298dac92327c47d62e3"} Jul 31 18:52:12.360885 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:12.360863 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 31 18:52:13.334715 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:13.334671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" event={"ID":"8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935","Type":"ContainerStarted","Data":"7e8ea9b2c888cd6e380f3524a2bd8db6b147f43785f7fb1c2d1fbe63a0223132"} Jul 31 18:52:13.334875 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:13.334839 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:13.354506 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:13.354432 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" podStartSLOduration=2.3440970500000002 podStartE2EDuration="7.35441588s" totalImagesPullingTime="5.01031883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:52:13.352813046 +0000 UTC m=+481.891040696" watchObservedRunningTime="2026-07-31 18:52:13.35441588 +0000 UTC m=+481.892643531" Jul 31 18:52:19.371144 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.371111 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xtszm" Jul 31 18:52:19.894585 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.894555 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:19.900384 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.900362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:19.903524 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.903479 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 18:52:19.905210 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.905176 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s7w92\"" Jul 31 18:52:19.905334 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.905215 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 18:52:19.907513 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.907361 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:19.923107 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:19.923084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t8l\" (UniqueName: \"kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l\") pod \"authorino-5dc7ccb5f-r84jv\" (UID: \"22368ee1-844f-4f2c-8567-49d3f5ab595e\") " pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:20.023734 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.023703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t8l\" (UniqueName: \"kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l\") pod \"authorino-5dc7ccb5f-r84jv\" (UID: \"22368ee1-844f-4f2c-8567-49d3f5ab595e\") " pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:20.038950 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.038927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t8l\" (UniqueName: \"kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l\") pod \"authorino-5dc7ccb5f-r84jv\" (UID: \"22368ee1-844f-4f2c-8567-49d3f5ab595e\") " pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:20.166776 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.166696 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:20.166971 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.166959 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:20.202081 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.201982 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64b57ff95f-ksprz"] Jul 31 18:52:20.207935 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.207911 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.213880 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.213849 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b57ff95f-ksprz"] Jul 31 18:52:20.226585 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.226365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjksr\" (UniqueName: \"kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr\") pod \"authorino-64b57ff95f-ksprz\" (UID: \"145ef44f-ad60-4db8-a8d5-4f78f02c5ecf\") " pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.258629 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.258595 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b57ff95f-ksprz"] Jul 31 18:52:20.258974 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:52:20.258945 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xjksr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-64b57ff95f-ksprz" podUID="145ef44f-ad60-4db8-a8d5-4f78f02c5ecf" Jul 31 18:52:20.289383 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.289353 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:52:20.293844 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.293823 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.296978 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.296958 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 31 18:52:20.302262 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.302241 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:52:20.307704 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.307686 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:20.309815 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:20.309781 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22368ee1_844f_4f2c_8567_49d3f5ab595e.slice/crio-820c5887ba519eda9329d15ee996c13d14e078381901ba020dfb22f0fac6d815 WatchSource:0}: Error finding container 820c5887ba519eda9329d15ee996c13d14e078381901ba020dfb22f0fac6d815: Status 404 returned error can't find the container with id 820c5887ba519eda9329d15ee996c13d14e078381901ba020dfb22f0fac6d815 Jul 31 18:52:20.327394 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.327370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjksr\" (UniqueName: \"kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr\") pod \"authorino-64b57ff95f-ksprz\" (UID: \"145ef44f-ad60-4db8-a8d5-4f78f02c5ecf\") " pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.327525 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.327441 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhq59\" (UniqueName: \"kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.327525 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.327464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.337012 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.336990 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjksr\" (UniqueName: \"kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr\") pod \"authorino-64b57ff95f-ksprz\" (UID: \"145ef44f-ad60-4db8-a8d5-4f78f02c5ecf\") " pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.371241 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.371208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" event={"ID":"22368ee1-844f-4f2c-8567-49d3f5ab595e","Type":"ContainerStarted","Data":"820c5887ba519eda9329d15ee996c13d14e078381901ba020dfb22f0fac6d815"} Jul 31 18:52:20.371616 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.371225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.378877 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.378853 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:20.428348 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.428269 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjksr\" (UniqueName: \"kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr\") pod \"145ef44f-ad60-4db8-a8d5-4f78f02c5ecf\" (UID: \"145ef44f-ad60-4db8-a8d5-4f78f02c5ecf\") " Jul 31 18:52:20.428673 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.428474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhq59\" (UniqueName: \"kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.428673 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.428533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.430481 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.430454 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr" (OuterVolumeSpecName: "kube-api-access-xjksr") pod "145ef44f-ad60-4db8-a8d5-4f78f02c5ecf" (UID: "145ef44f-ad60-4db8-a8d5-4f78f02c5ecf"). InnerVolumeSpecName "kube-api-access-xjksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:52:20.430980 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.430963 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.438401 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.438377 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhq59\" (UniqueName: \"kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59\") pod \"authorino-5ccc6c4677-xd4x8\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.529173 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.529141 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjksr\" (UniqueName: \"kubernetes.io/projected/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf-kube-api-access-xjksr\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:20.603601 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.603575 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:52:20.734269 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:20.734247 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:52:20.736103 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:20.736072 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ac7e41c_58d0_48ce_bdd8_f10efc2c79d2.slice/crio-92b45e68a6165041ae36cb60b39208298791d3e7a205933bcb71d34883271d18 WatchSource:0}: Error finding container 92b45e68a6165041ae36cb60b39208298791d3e7a205933bcb71d34883271d18: Status 404 returned error can't find the container with id 92b45e68a6165041ae36cb60b39208298791d3e7a205933bcb71d34883271d18 Jul 31 18:52:21.377847 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.377809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" event={"ID":"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2","Type":"ContainerStarted","Data":"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445"} Jul 31 18:52:21.377847 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.377836 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b57ff95f-ksprz" Jul 31 18:52:21.378420 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.377863 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" event={"ID":"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2","Type":"ContainerStarted","Data":"92b45e68a6165041ae36cb60b39208298791d3e7a205933bcb71d34883271d18"} Jul 31 18:52:21.402574 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.402517 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" podStartSLOduration=0.856431841 podStartE2EDuration="1.402483795s" totalImagesPullingTime="546.051954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:52:21.402052929 +0000 UTC m=+489.940280561" watchObservedRunningTime="2026-07-31 18:52:21.402483795 +0000 UTC m=+489.940711447" Jul 31 18:52:21.431157 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.430949 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b57ff95f-ksprz"] Jul 31 18:52:21.435282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:21.435246 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64b57ff95f-ksprz"] Jul 31 18:52:22.034668 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.034592 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145ef44f-ad60-4db8-a8d5-4f78f02c5ecf" path="/var/lib/kubelet/pods/145ef44f-ad60-4db8-a8d5-4f78f02c5ecf/volumes" Jul 31 18:52:22.384349 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.384260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" event={"ID":"22368ee1-844f-4f2c-8567-49d3f5ab595e","Type":"ContainerStarted","Data":"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3"} Jul 31 18:52:22.384349 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.384300 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" podUID="22368ee1-844f-4f2c-8567-49d3f5ab595e" containerName="authorino" containerID="cri-o://85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3" gracePeriod=30 Jul 31 18:52:22.403904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.403857 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" podStartSLOduration=2.041038754 podStartE2EDuration="3.403839536s" totalImagesPullingTime="1.362800782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:52:22.403295759 +0000 UTC m=+490.941523410" watchObservedRunningTime="2026-07-31 18:52:22.403839536 +0000 UTC m=+490.942067187" Jul 31 18:52:22.644391 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.644368 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:22.749603 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.749572 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9t8l\" (UniqueName: \"kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l\") pod \"22368ee1-844f-4f2c-8567-49d3f5ab595e\" (UID: \"22368ee1-844f-4f2c-8567-49d3f5ab595e\") " Jul 31 18:52:22.751838 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.751811 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l" (OuterVolumeSpecName: "kube-api-access-m9t8l") pod "22368ee1-844f-4f2c-8567-49d3f5ab595e" (UID: "22368ee1-844f-4f2c-8567-49d3f5ab595e"). InnerVolumeSpecName "kube-api-access-m9t8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:52:22.850093 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:22.850056 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9t8l\" (UniqueName: \"kubernetes.io/projected/22368ee1-844f-4f2c-8567-49d3f5ab595e-kube-api-access-m9t8l\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:23.390186 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.390154 2566 generic.go:356] "Generic (PLEG): container finished" podID="22368ee1-844f-4f2c-8567-49d3f5ab595e" containerID="85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3" exitCode=0 Jul 31 18:52:23.390641 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.390194 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" Jul 31 18:52:23.390641 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.390194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" event={"ID":"22368ee1-844f-4f2c-8567-49d3f5ab595e","Type":"ContainerDied","Data":"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3"} Jul 31 18:52:23.390641 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.390237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-r84jv" event={"ID":"22368ee1-844f-4f2c-8567-49d3f5ab595e","Type":"ContainerDied","Data":"820c5887ba519eda9329d15ee996c13d14e078381901ba020dfb22f0fac6d815"} Jul 31 18:52:23.390641 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.390257 2566 scope.go:117] "RemoveContainer" containerID="85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3" Jul 31 18:52:23.398974 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.398958 2566 scope.go:117] "RemoveContainer" containerID="85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3" Jul 31 18:52:23.399215 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:52:23.399198 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3\": container with ID starting with 85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3 not found: ID does not exist" containerID="85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3" Jul 31 18:52:23.399259 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.399224 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3"} err="failed to get container status \"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3\": rpc error: code = NotFound desc = could not find container \"85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3\": container with ID starting with 85582cfe27a6c719f11af4bae7ad3c692d71d0b13ba715f2c077f6e32d2997c3 not found: ID does not exist" Jul 31 18:52:23.413681 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.413658 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:23.417600 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:23.417577 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-r84jv"] Jul 31 18:52:24.036638 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:24.036605 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22368ee1-844f-4f2c-8567-49d3f5ab595e" path="/var/lib/kubelet/pods/22368ee1-844f-4f2c-8567-49d3f5ab595e/volumes" Jul 31 18:52:26.100845 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.100808 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:26.101213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.101200 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22368ee1-844f-4f2c-8567-49d3f5ab595e" containerName="authorino" Jul 31 18:52:26.101258 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.101215 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22368ee1-844f-4f2c-8567-49d3f5ab595e" containerName="authorino" Jul 31 18:52:26.101292 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.101281 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22368ee1-844f-4f2c-8567-49d3f5ab595e" containerName="authorino" Jul 31 18:52:26.223225 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.223187 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:26.223399 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.223322 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.227128 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.227104 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 18:52:26.227267 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.227126 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 18:52:26.227267 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.227134 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5chrg\"" Jul 31 18:52:26.279333 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.279296 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:52:26.281790 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.281768 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.282121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.282102 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdd4\" (UniqueName: \"kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.282184 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.282138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.295216 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.295189 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:52:26.383289 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.383205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.383289 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.383262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmvq\" (UniqueName: \"kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.383463 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.383328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdd4\" (UniqueName: \"kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.383463 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.383364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.385942 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.385917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.395597 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.395577 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdd4\" (UniqueName: \"kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4\") pod \"maas-controller-694fdffc9b-4tmpt\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.484771 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.484736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.484928 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.484794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmvq\" (UniqueName: \"kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.487384 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.487349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.497206 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.497182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmvq\" (UniqueName: \"kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq\") pod \"maas-controller-5667fb5f9d-9t58q\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.533093 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.533064 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:26.591454 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.591415 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:26.671390 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.671360 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:26.672646 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:26.672619 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded19b82d_d366_4eb3_8fe0_7eb639702299.slice/crio-995ecf9b77ac7ef9a0e11dea2c11c5b3fe7f9ac7397ba66b9972e10702dbeedc WatchSource:0}: Error finding container 995ecf9b77ac7ef9a0e11dea2c11c5b3fe7f9ac7397ba66b9972e10702dbeedc: Status 404 returned error can't find the container with id 995ecf9b77ac7ef9a0e11dea2c11c5b3fe7f9ac7397ba66b9972e10702dbeedc Jul 31 18:52:26.738664 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:26.738637 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:52:26.741368 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:26.741340 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3e303b_465e_4b54_bf3f_accd58814351.slice/crio-a601d348d240f1ea0b4a80c846bc8f32249a54e6c1d835ae55638ed62995b1f5 WatchSource:0}: Error finding container a601d348d240f1ea0b4a80c846bc8f32249a54e6c1d835ae55638ed62995b1f5: Status 404 returned error can't find the container with id a601d348d240f1ea0b4a80c846bc8f32249a54e6c1d835ae55638ed62995b1f5 Jul 31 18:52:27.428822 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:27.428737 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" event={"ID":"ed19b82d-d366-4eb3-8fe0-7eb639702299","Type":"ContainerStarted","Data":"995ecf9b77ac7ef9a0e11dea2c11c5b3fe7f9ac7397ba66b9972e10702dbeedc"} Jul 31 18:52:27.433205 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:27.433173 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" event={"ID":"8d3e303b-465e-4b54-bf3f-accd58814351","Type":"ContainerStarted","Data":"a601d348d240f1ea0b4a80c846bc8f32249a54e6c1d835ae55638ed62995b1f5"} Jul 31 18:52:29.444012 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.443913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" event={"ID":"8d3e303b-465e-4b54-bf3f-accd58814351","Type":"ContainerStarted","Data":"8e1673657b50e019f204a796ebbe0929b01e614103ad5a10d1d99dfe7013a526"} Jul 31 18:52:29.444012 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.443975 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:29.445319 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.445286 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" event={"ID":"ed19b82d-d366-4eb3-8fe0-7eb639702299","Type":"ContainerStarted","Data":"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671"} Jul 31 18:52:29.445426 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.445332 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:29.470656 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.470607 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" podStartSLOduration=1.043664557 podStartE2EDuration="3.47059433s" totalImagesPullingTime="2.426929773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:52:29.467740203 +0000 UTC m=+498.005967856" watchObservedRunningTime="2026-07-31 18:52:29.47059433 +0000 UTC m=+498.008821981" Jul 31 18:52:29.492973 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:29.492933 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" podStartSLOduration=1.007298691 podStartE2EDuration="3.492918988s" totalImagesPullingTime="2.485620297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:52:29.491019567 +0000 UTC m=+498.029247217" watchObservedRunningTime="2026-07-31 18:52:29.492918988 +0000 UTC m=+498.031146645" Jul 31 18:52:40.452031 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.452001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:52:40.452406 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.452066 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:40.526755 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.526718 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:40.526984 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.526957 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" podUID="ed19b82d-d366-4eb3-8fe0-7eb639702299" containerName="manager" containerID="cri-o://dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671" gracePeriod=30 Jul 31 18:52:40.767838 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.767811 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:40.924731 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.924690 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert\") pod \"ed19b82d-d366-4eb3-8fe0-7eb639702299\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " Jul 31 18:52:40.924921 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.924767 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdd4\" (UniqueName: \"kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4\") pod \"ed19b82d-d366-4eb3-8fe0-7eb639702299\" (UID: \"ed19b82d-d366-4eb3-8fe0-7eb639702299\") " Jul 31 18:52:40.927140 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.927113 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4" (OuterVolumeSpecName: "kube-api-access-qmdd4") pod "ed19b82d-d366-4eb3-8fe0-7eb639702299" (UID: "ed19b82d-d366-4eb3-8fe0-7eb639702299"). InnerVolumeSpecName "kube-api-access-qmdd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:52:40.927140 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:40.927124 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ed19b82d-d366-4eb3-8fe0-7eb639702299" (UID: "ed19b82d-d366-4eb3-8fe0-7eb639702299"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:52:41.026271 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.026180 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed19b82d-d366-4eb3-8fe0-7eb639702299-webhook-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:41.026271 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.026225 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdd4\" (UniqueName: \"kubernetes.io/projected/ed19b82d-d366-4eb3-8fe0-7eb639702299-kube-api-access-qmdd4\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:52:41.502831 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.502795 2566 generic.go:356] "Generic (PLEG): container finished" podID="ed19b82d-d366-4eb3-8fe0-7eb639702299" containerID="dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671" exitCode=0 Jul 31 18:52:41.503213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.502837 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" Jul 31 18:52:41.503213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.502881 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" event={"ID":"ed19b82d-d366-4eb3-8fe0-7eb639702299","Type":"ContainerDied","Data":"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671"} Jul 31 18:52:41.503213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.502934 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694fdffc9b-4tmpt" event={"ID":"ed19b82d-d366-4eb3-8fe0-7eb639702299","Type":"ContainerDied","Data":"995ecf9b77ac7ef9a0e11dea2c11c5b3fe7f9ac7397ba66b9972e10702dbeedc"} Jul 31 18:52:41.503213 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.502960 2566 scope.go:117] "RemoveContainer" containerID="dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671" Jul 31 18:52:41.511933 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.511918 2566 scope.go:117] "RemoveContainer" containerID="dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671" Jul 31 18:52:41.512179 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:52:41.512163 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671\": container with ID starting with dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671 not found: ID does not exist" containerID="dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671" Jul 31 18:52:41.512228 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.512186 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671"} err="failed to get container status \"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671\": rpc error: code = NotFound desc = could not find container \"dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671\": container with ID starting with dce1dce226511a1a468ca92b39fe095cfc66ba2a208048b6d93271c1f3e03671 not found: ID does not exist" Jul 31 18:52:41.527078 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.527053 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:41.531684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:41.531664 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-694fdffc9b-4tmpt"] Jul 31 18:52:42.035121 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:42.035089 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed19b82d-d366-4eb3-8fe0-7eb639702299" path="/var/lib/kubelet/pods/ed19b82d-d366-4eb3-8fe0-7eb639702299/volumes" Jul 31 18:52:58.497639 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.497601 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q"] Jul 31 18:52:58.498226 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.498205 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed19b82d-d366-4eb3-8fe0-7eb639702299" containerName="manager" Jul 31 18:52:58.498226 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.498225 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed19b82d-d366-4eb3-8fe0-7eb639702299" containerName="manager" Jul 31 18:52:58.498348 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.498332 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed19b82d-d366-4eb3-8fe0-7eb639702299" containerName="manager" Jul 31 18:52:58.502418 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.502397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.506716 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.506696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\"" Jul 31 18:52:58.506844 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.506697 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 31 18:52:58.511498 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.511471 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q"] Jul 31 18:52:58.605226 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.605182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-maas-api-tls\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.605421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.605247 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npptp\" (UniqueName: \"kubernetes.io/projected/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-kube-api-access-npptp\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.605421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.605283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-openshift-service-ca\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.706566 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.706529 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-maas-api-tls\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.706770 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.706580 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npptp\" (UniqueName: \"kubernetes.io/projected/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-kube-api-access-npptp\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.706770 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.706606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-openshift-service-ca\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.707251 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.707232 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-openshift-service-ca\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.709151 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.709124 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-maas-api-tls\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.718680 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.718656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npptp\" (UniqueName: \"kubernetes.io/projected/7bb9cd45-7a31-4785-b24f-9353ca06fa3c-kube-api-access-npptp\") pod \"maas-api-7c89d6c77-dqb6q\" (UID: \"7bb9cd45-7a31-4785-b24f-9353ca06fa3c\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.813598 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.813482 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:52:58.957548 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.957522 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q"] Jul 31 18:52:58.959756 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:58.959728 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb9cd45_7a31_4785_b24f_9353ca06fa3c.slice/crio-b8d5f539a53f954d900c8d883a8f1c21e0ceb3eb5e2b8f90fa9c7a0caaf4bfe5 WatchSource:0}: Error finding container b8d5f539a53f954d900c8d883a8f1c21e0ceb3eb5e2b8f90fa9c7a0caaf4bfe5: Status 404 returned error can't find the container with id b8d5f539a53f954d900c8d883a8f1c21e0ceb3eb5e2b8f90fa9c7a0caaf4bfe5 Jul 31 18:52:58.998146 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:58.998105 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj"] Jul 31 18:52:59.002892 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.002870 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.005821 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.005802 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 31 18:52:59.012398 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.012372 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj"] Jul 31 18:52:59.027385 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.027365 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fl624"] Jul 31 18:52:59.030828 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.030809 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.033840 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.033822 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g9rd2\"" Jul 31 18:52:59.040860 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.040839 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fl624"] Jul 31 18:52:59.109914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.109826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89da6e1-91b4-49a2-84f6-b89ac08918c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.109914 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.109872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997605d3-a1e0-4855-a57b-5294c87a4efb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.110150 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.110007 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4mt\" (UniqueName: \"kubernetes.io/projected/f89da6e1-91b4-49a2-84f6-b89ac08918c6-kube-api-access-vn4mt\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.110150 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.110092 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k8wb\" (UniqueName: \"kubernetes.io/projected/997605d3-a1e0-4855-a57b-5294c87a4efb-kube-api-access-4k8wb\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.210813 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.210783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4mt\" (UniqueName: \"kubernetes.io/projected/f89da6e1-91b4-49a2-84f6-b89ac08918c6-kube-api-access-vn4mt\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.210985 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.210843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k8wb\" (UniqueName: \"kubernetes.io/projected/997605d3-a1e0-4855-a57b-5294c87a4efb-kube-api-access-4k8wb\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.210985 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.210876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89da6e1-91b4-49a2-84f6-b89ac08918c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.210985 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.210900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997605d3-a1e0-4855-a57b-5294c87a4efb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.211625 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.211604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997605d3-a1e0-4855-a57b-5294c87a4efb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.211765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.211742 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89da6e1-91b4-49a2-84f6-b89ac08918c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.220844 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.220817 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4mt\" (UniqueName: \"kubernetes.io/projected/f89da6e1-91b4-49a2-84f6-b89ac08918c6-kube-api-access-vn4mt\") pod \"payload-processing-5cc6cd444b-fl624\" (UID: \"f89da6e1-91b4-49a2-84f6-b89ac08918c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.220951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.220818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k8wb\" (UniqueName: \"kubernetes.io/projected/997605d3-a1e0-4855-a57b-5294c87a4efb-kube-api-access-4k8wb\") pod \"payload-pre-processing-687dbb6cf6-fmczj\" (UID: \"997605d3-a1e0-4855-a57b-5294c87a4efb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.314203 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.314172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:52:59.341814 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.341786 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:52:59.464883 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.464856 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj"] Jul 31 18:52:59.468928 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:59.468884 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997605d3_a1e0_4855_a57b_5294c87a4efb.slice/crio-7b27eb2c62456039d0b22db640a99a3debc97012a430c8ed91f7aa99151504f7 WatchSource:0}: Error finding container 7b27eb2c62456039d0b22db640a99a3debc97012a430c8ed91f7aa99151504f7: Status 404 returned error can't find the container with id 7b27eb2c62456039d0b22db640a99a3debc97012a430c8ed91f7aa99151504f7 Jul 31 18:52:59.512162 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.512138 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-fl624"] Jul 31 18:52:59.513890 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:52:59.513862 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89da6e1_91b4_49a2_84f6_b89ac08918c6.slice/crio-95aae7c924ac1e668f08f325436865787d8117f7b333026c8d344a27f3ada0ae WatchSource:0}: Error finding container 95aae7c924ac1e668f08f325436865787d8117f7b333026c8d344a27f3ada0ae: Status 404 returned error can't find the container with id 95aae7c924ac1e668f08f325436865787d8117f7b333026c8d344a27f3ada0ae Jul 31 18:52:59.590113 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.590078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" event={"ID":"997605d3-a1e0-4855-a57b-5294c87a4efb","Type":"ContainerStarted","Data":"7b27eb2c62456039d0b22db640a99a3debc97012a430c8ed91f7aa99151504f7"} Jul 31 18:52:59.591282 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.591249 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" event={"ID":"7bb9cd45-7a31-4785-b24f-9353ca06fa3c","Type":"ContainerStarted","Data":"b8d5f539a53f954d900c8d883a8f1c21e0ceb3eb5e2b8f90fa9c7a0caaf4bfe5"} Jul 31 18:52:59.592328 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:52:59.592311 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" event={"ID":"f89da6e1-91b4-49a2-84f6-b89ac08918c6","Type":"ContainerStarted","Data":"95aae7c924ac1e668f08f325436865787d8117f7b333026c8d344a27f3ada0ae"} Jul 31 18:53:01.610553 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.610508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" event={"ID":"f89da6e1-91b4-49a2-84f6-b89ac08918c6","Type":"ContainerStarted","Data":"8543088f36a557f5fadda5886d98d7150ad3714b16be96ee1e6f99b008b9e60c"} Jul 31 18:53:01.611024 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.610659 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:53:01.612415 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.612384 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" event={"ID":"997605d3-a1e0-4855-a57b-5294c87a4efb","Type":"ContainerStarted","Data":"49c7af29e5418526f59cd10766c9d94086dd374d3a78fef4c41142361218333a"} Jul 31 18:53:01.612579 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.612478 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:53:01.614107 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.614088 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" event={"ID":"7bb9cd45-7a31-4785-b24f-9353ca06fa3c","Type":"ContainerStarted","Data":"178b3e8f3a294b1075eb99893c75ed3c9131e28749e7aa161b0c90784db26f10"} Jul 31 18:53:01.614289 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.614272 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:53:01.634290 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.634233 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" podStartSLOduration=0.714007952 podStartE2EDuration="2.634213384s" totalImagesPullingTime="1.920205432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:01.631872486 +0000 UTC m=+530.170100141" watchObservedRunningTime="2026-07-31 18:53:01.634213384 +0000 UTC m=+530.172441038" Jul 31 18:53:01.653773 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.653714 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" podStartSLOduration=1.699212477 podStartE2EDuration="3.653698826s" totalImagesPullingTime="1.954486349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:01.651905429 +0000 UTC m=+530.190133091" watchObservedRunningTime="2026-07-31 18:53:01.653698826 +0000 UTC m=+530.191926454" Jul 31 18:53:01.671945 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:01.671844 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" podStartSLOduration=1.20787797 podStartE2EDuration="3.671831826s" totalImagesPullingTime="2.463953856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:01.671416907 +0000 UTC m=+530.209644557" watchObservedRunningTime="2026-07-31 18:53:01.671831826 +0000 UTC m=+530.210059477" Jul 31 18:53:07.625802 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:07.625773 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-dqb6q" Jul 31 18:53:12.622409 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:12.622321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-fl624" Jul 31 18:53:12.622409 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:12.622390 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fmczj" Jul 31 18:53:22.483025 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.482977 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92"] Jul 31 18:53:22.487869 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.487844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.491227 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.491204 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 31 18:53:22.491338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.491253 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 18:53:22.491338 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.491270 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 18:53:22.492745 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.492728 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs6tj\"" Jul 31 18:53:22.498014 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.497992 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92"] Jul 31 18:53:22.633092 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.633295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.633295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633163 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzb4l\" (UniqueName: \"kubernetes.io/projected/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-kube-api-access-nzb4l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.633295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633201 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.633295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633223 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.633295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.633246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.734778 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.734778 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.734778 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734776 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzb4l\" (UniqueName: \"kubernetes.io/projected/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-kube-api-access-nzb4l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735063 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735063 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735063 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.734866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735208 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.735147 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735208 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.735160 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.735304 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.735210 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.737227 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.737206 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.737574 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.737555 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.744207 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.744185 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzb4l\" (UniqueName: \"kubernetes.io/projected/dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f-kube-api-access-nzb4l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mmb92\" (UID: \"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.800259 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.800226 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:22.940321 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:22.940298 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92"] Jul 31 18:53:22.942284 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:53:22.942256 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7d6ec6_5e3f_4cd1_bdcb_191e36f59a4f.slice/crio-43e75352d255311a9eed1f2b24e49a455501b6bbe706c03900cfaf0877f1e5ca WatchSource:0}: Error finding container 43e75352d255311a9eed1f2b24e49a455501b6bbe706c03900cfaf0877f1e5ca: Status 404 returned error can't find the container with id 43e75352d255311a9eed1f2b24e49a455501b6bbe706c03900cfaf0877f1e5ca Jul 31 18:53:23.724162 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:23.724130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" event={"ID":"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f","Type":"ContainerStarted","Data":"43e75352d255311a9eed1f2b24e49a455501b6bbe706c03900cfaf0877f1e5ca"} Jul 31 18:53:24.730506 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:24.730467 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" event={"ID":"dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f","Type":"ContainerStarted","Data":"15706e7b78940150f35950a7e48267bb258ae6c90a7f074bd99a02a4c7ce1e5a"} Jul 31 18:53:24.751065 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:24.751019 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" podStartSLOduration=1.564063452 podStartE2EDuration="2.751004789s" totalImagesPullingTime="1.186941337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:24.749823751 +0000 UTC m=+553.288051425" watchObservedRunningTime="2026-07-31 18:53:24.751004789 +0000 UTC m=+553.289232482" Jul 31 18:53:30.480995 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.480956 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2"] Jul 31 18:53:30.485310 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.485289 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.488279 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.488258 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 31 18:53:30.498737 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.498715 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2"] Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503304 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2qn\" (UniqueName: \"kubernetes.io/projected/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-kube-api-access-ns2qn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.503645 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.503608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604446 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604446 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604456 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2qn\" (UniqueName: \"kubernetes.io/projected/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-kube-api-access-ns2qn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604742 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.604997 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.604972 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.605099 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.605034 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.605140 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.605123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.607053 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.607031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.607285 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.607261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.614268 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.614239 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2qn\" (UniqueName: \"kubernetes.io/projected/ffcd3813-4be7-4e3b-ba23-c639ce1eb702-kube-api-access-ns2qn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf6b2\" (UID: \"ffcd3813-4be7-4e3b-ba23-c639ce1eb702\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.796531 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.796411 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:30.931183 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:30.931160 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2"] Jul 31 18:53:30.933440 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:53:30.933413 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcd3813_4be7_4e3b_ba23_c639ce1eb702.slice/crio-966af8dfc22126e475b4598cd498cf08dbc323e04acc077bc8a0d337fe49abad WatchSource:0}: Error finding container 966af8dfc22126e475b4598cd498cf08dbc323e04acc077bc8a0d337fe49abad: Status 404 returned error can't find the container with id 966af8dfc22126e475b4598cd498cf08dbc323e04acc077bc8a0d337fe49abad Jul 31 18:53:31.767834 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:31.767795 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" event={"ID":"ffcd3813-4be7-4e3b-ba23-c639ce1eb702","Type":"ContainerStarted","Data":"9644df0caa5b56f7e10a18f009070df3847308eeaca16cc9c5367ef387fddd3d"} Jul 31 18:53:31.768230 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:31.767853 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" event={"ID":"ffcd3813-4be7-4e3b-ba23-c639ce1eb702","Type":"ContainerStarted","Data":"966af8dfc22126e475b4598cd498cf08dbc323e04acc077bc8a0d337fe49abad"} Jul 31 18:53:31.790639 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:31.790587 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" podStartSLOduration=1.5217056850000001 podStartE2EDuration="1.790566751s" totalImagesPullingTime="268.861066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:31.788546263 +0000 UTC m=+560.326773915" watchObservedRunningTime="2026-07-31 18:53:31.790566751 +0000 UTC m=+560.328794401" Jul 31 18:53:32.800628 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:32.800581 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:32.801080 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:32.800710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:32.813622 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:32.813594 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:33.789407 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:33.789375 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mmb92" Jul 31 18:53:40.797183 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:40.797141 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:40.797712 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:40.797216 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:40.809656 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:40.809626 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:53:40.824735 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:53:40.824711 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf6b2" Jul 31 18:55:30.225477 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.225440 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:55:30.226745 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.226626 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" podUID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" containerName="authorino" containerID="cri-o://9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445" gracePeriod=30 Jul 31 18:55:30.481064 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.481013 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:55:30.534328 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.534301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert\") pod \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " Jul 31 18:55:30.534483 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.534364 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhq59\" (UniqueName: \"kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59\") pod \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\" (UID: \"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2\") " Jul 31 18:55:30.536503 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.536462 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59" (OuterVolumeSpecName: "kube-api-access-lhq59") pod "8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" (UID: "8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2"). InnerVolumeSpecName "kube-api-access-lhq59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:55:30.545097 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.545072 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" (UID: "8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:55:30.636003 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.635974 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-tls-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:55:30.636003 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:30.636003 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhq59\" (UniqueName: \"kubernetes.io/projected/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2-kube-api-access-lhq59\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:55:31.344534 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.344470 2566 generic.go:356] "Generic (PLEG): container finished" podID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" containerID="9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445" exitCode=0 Jul 31 18:55:31.345004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.344545 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" Jul 31 18:55:31.345004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.344587 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" event={"ID":"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2","Type":"ContainerDied","Data":"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445"} Jul 31 18:55:31.345004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.344616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6c4677-xd4x8" event={"ID":"8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2","Type":"ContainerDied","Data":"92b45e68a6165041ae36cb60b39208298791d3e7a205933bcb71d34883271d18"} Jul 31 18:55:31.345004 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.344631 2566 scope.go:117] "RemoveContainer" containerID="9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445" Jul 31 18:55:31.354566 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.354545 2566 scope.go:117] "RemoveContainer" containerID="9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445" Jul 31 18:55:31.354822 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:55:31.354801 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445\": container with ID starting with 9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445 not found: ID does not exist" containerID="9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445" Jul 31 18:55:31.354904 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.354831 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445"} err="failed to get container status \"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445\": rpc error: code = NotFound desc = could not find container \"9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445\": container with ID starting with 9ce8a811cc1549958a1970702d676d3cff66a0a3a90ee852ba280c0b4f5b2445 not found: ID does not exist" Jul 31 18:55:31.371295 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.371272 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:55:31.374650 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:31.374631 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ccc6c4677-xd4x8"] Jul 31 18:55:32.036951 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:55:32.036920 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" path="/var/lib/kubelet/pods/8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2/volumes" Jul 31 18:56:17.340777 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.340743 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:56:17.341264 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.340956 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" podUID="8d3e303b-465e-4b54-bf3f-accd58814351" containerName="manager" containerID="cri-o://8e1673657b50e019f204a796ebbe0929b01e614103ad5a10d1d99dfe7013a526" gracePeriod=30 Jul 31 18:56:17.569468 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.569433 2566 generic.go:356] "Generic (PLEG): container finished" podID="8d3e303b-465e-4b54-bf3f-accd58814351" containerID="8e1673657b50e019f204a796ebbe0929b01e614103ad5a10d1d99dfe7013a526" exitCode=0 Jul 31 18:56:17.569649 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.569468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" event={"ID":"8d3e303b-465e-4b54-bf3f-accd58814351","Type":"ContainerDied","Data":"8e1673657b50e019f204a796ebbe0929b01e614103ad5a10d1d99dfe7013a526"} Jul 31 18:56:17.588453 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.588428 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:56:17.677979 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.677954 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert\") pod \"8d3e303b-465e-4b54-bf3f-accd58814351\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " Jul 31 18:56:17.678180 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.677996 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmvq\" (UniqueName: \"kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq\") pod \"8d3e303b-465e-4b54-bf3f-accd58814351\" (UID: \"8d3e303b-465e-4b54-bf3f-accd58814351\") " Jul 31 18:56:17.680125 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.680099 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq" (OuterVolumeSpecName: "kube-api-access-jfmvq") pod "8d3e303b-465e-4b54-bf3f-accd58814351" (UID: "8d3e303b-465e-4b54-bf3f-accd58814351"). InnerVolumeSpecName "kube-api-access-jfmvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:56:17.680223 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.680187 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8d3e303b-465e-4b54-bf3f-accd58814351" (UID: "8d3e303b-465e-4b54-bf3f-accd58814351"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:56:17.778814 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.778782 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d3e303b-465e-4b54-bf3f-accd58814351-webhook-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:56:17.778814 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:17.778808 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmvq\" (UniqueName: \"kubernetes.io/projected/8d3e303b-465e-4b54-bf3f-accd58814351-kube-api-access-jfmvq\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:56:18.575017 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:18.574987 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" Jul 31 18:56:18.575421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:18.575034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-9t58q" event={"ID":"8d3e303b-465e-4b54-bf3f-accd58814351","Type":"ContainerDied","Data":"a601d348d240f1ea0b4a80c846bc8f32249a54e6c1d835ae55638ed62995b1f5"} Jul 31 18:56:18.575421 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:18.575100 2566 scope.go:117] "RemoveContainer" containerID="8e1673657b50e019f204a796ebbe0929b01e614103ad5a10d1d99dfe7013a526" Jul 31 18:56:18.597092 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:18.597069 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:56:18.606667 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:18.606644 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-9t58q"] Jul 31 18:56:20.037524 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:56:20.036645 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d3e303b-465e-4b54-bf3f-accd58814351" path="/var/lib/kubelet/pods/8d3e303b-465e-4b54-bf3f-accd58814351/volumes" Jul 31 18:57:19.393026 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.392995 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:19.393455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393392 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" containerName="authorino" Jul 31 18:57:19.393455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393403 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" containerName="authorino" Jul 31 18:57:19.393455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393428 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3e303b-465e-4b54-bf3f-accd58814351" containerName="manager" Jul 31 18:57:19.393455 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393433 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3e303b-465e-4b54-bf3f-accd58814351" containerName="manager" Jul 31 18:57:19.393617 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393530 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac7e41c-58d0-48ce-bdd8-f10efc2c79d2" containerName="authorino" Jul 31 18:57:19.393617 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.393539 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3e303b-465e-4b54-bf3f-accd58814351" containerName="manager" Jul 31 18:57:19.396741 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.396726 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.402033 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.402001 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 18:57:19.402161 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.402037 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 18:57:19.402161 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.402069 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5chrg\"" Jul 31 18:57:19.409970 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.409949 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:19.560037 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.560003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8p2p\" (UniqueName: \"kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.560198 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.560066 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.660927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.660848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8p2p\" (UniqueName: \"kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.660927 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.660923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.663407 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.663389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.675900 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.675869 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8p2p\" (UniqueName: \"kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p\") pod \"maas-controller-5667fb5f9d-7h4t5\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.706355 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.706329 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:19.850659 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.850632 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:19.853049 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:57:19.853028 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a0b5d8_4493_4235_b051_14aa7183933d.slice/crio-b429710f4739c44cef1ff9cdd0919f99357f03c2c1f4b3ea4b7741cfed0579db WatchSource:0}: Error finding container b429710f4739c44cef1ff9cdd0919f99357f03c2c1f4b3ea4b7741cfed0579db: Status 404 returned error can't find the container with id b429710f4739c44cef1ff9cdd0919f99357f03c2c1f4b3ea4b7741cfed0579db Jul 31 18:57:19.854388 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.854370 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 18:57:19.859027 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:19.859001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" event={"ID":"21a0b5d8-4493-4235-b051-14aa7183933d","Type":"ContainerStarted","Data":"b429710f4739c44cef1ff9cdd0919f99357f03c2c1f4b3ea4b7741cfed0579db"} Jul 31 18:57:20.868828 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:20.868794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" event={"ID":"21a0b5d8-4493-4235-b051-14aa7183933d","Type":"ContainerStarted","Data":"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab"} Jul 31 18:57:20.869223 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:20.868924 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:20.906960 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:20.906911 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" podStartSLOduration=1.4706894240000001 podStartE2EDuration="1.906893889s" totalImagesPullingTime="436.204465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:57:20.905477896 +0000 UTC m=+789.443705559" watchObservedRunningTime="2026-07-31 18:57:20.906893889 +0000 UTC m=+789.445121603" Jul 31 18:57:31.876328 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:31.876296 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:36.499007 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.498971 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:36.499400 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.499175 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" podUID="21a0b5d8-4493-4235-b051-14aa7183933d" containerName="manager" containerID="cri-o://2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab" gracePeriod=30 Jul 31 18:57:36.741761 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.741739 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:36.820190 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.820114 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8p2p\" (UniqueName: \"kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p\") pod \"21a0b5d8-4493-4235-b051-14aa7183933d\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " Jul 31 18:57:36.820190 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.820176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert\") pod \"21a0b5d8-4493-4235-b051-14aa7183933d\" (UID: \"21a0b5d8-4493-4235-b051-14aa7183933d\") " Jul 31 18:57:36.822399 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.822369 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p" (OuterVolumeSpecName: "kube-api-access-z8p2p") pod "21a0b5d8-4493-4235-b051-14aa7183933d" (UID: "21a0b5d8-4493-4235-b051-14aa7183933d"). InnerVolumeSpecName "kube-api-access-z8p2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 18:57:36.822399 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.822382 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "21a0b5d8-4493-4235-b051-14aa7183933d" (UID: "21a0b5d8-4493-4235-b051-14aa7183933d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 18:57:36.921296 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.921264 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8p2p\" (UniqueName: \"kubernetes.io/projected/21a0b5d8-4493-4235-b051-14aa7183933d-kube-api-access-z8p2p\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:57:36.921296 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.921289 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21a0b5d8-4493-4235-b051-14aa7183933d-webhook-cert\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 18:57:36.959400 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.959368 2566 generic.go:356] "Generic (PLEG): container finished" podID="21a0b5d8-4493-4235-b051-14aa7183933d" containerID="2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab" exitCode=0 Jul 31 18:57:36.959599 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.959419 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" Jul 31 18:57:36.959599 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.959452 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" event={"ID":"21a0b5d8-4493-4235-b051-14aa7183933d","Type":"ContainerDied","Data":"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab"} Jul 31 18:57:36.959599 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.959525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-7h4t5" event={"ID":"21a0b5d8-4493-4235-b051-14aa7183933d","Type":"ContainerDied","Data":"b429710f4739c44cef1ff9cdd0919f99357f03c2c1f4b3ea4b7741cfed0579db"} Jul 31 18:57:36.959599 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.959550 2566 scope.go:117] "RemoveContainer" containerID="2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab" Jul 31 18:57:36.968684 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.968666 2566 scope.go:117] "RemoveContainer" containerID="2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab" Jul 31 18:57:36.968923 ip-10-0-139-43 kubenswrapper[2566]: E0731 18:57:36.968907 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab\": container with ID starting with 2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab not found: ID does not exist" containerID="2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab" Jul 31 18:57:36.968977 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.968930 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab"} err="failed to get container status \"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab\": rpc error: code = NotFound desc = could not find container \"2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab\": container with ID starting with 2cb5739cb07dcb989c7d8dc932815dedd7c045014571f010a9cefe33aa6518ab not found: ID does not exist" Jul 31 18:57:36.984372 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.984349 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:36.988725 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:36.988703 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-7h4t5"] Jul 31 18:57:37.765765 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.765732 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-clqzj"] Jul 31 18:57:37.766221 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.766156 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a0b5d8-4493-4235-b051-14aa7183933d" containerName="manager" Jul 31 18:57:37.766221 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.766168 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a0b5d8-4493-4235-b051-14aa7183933d" containerName="manager" Jul 31 18:57:37.766293 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.766233 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a0b5d8-4493-4235-b051-14aa7183933d" containerName="manager" Jul 31 18:57:37.770553 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.770536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.775101 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.775079 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5chrg\"" Jul 31 18:57:37.775236 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.775113 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 18:57:37.775236 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.775088 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 18:57:37.780135 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.780114 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-clqzj"] Jul 31 18:57:37.830884 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.830832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93253fe4-60ab-4ac5-9285-f122558b6da2-webhook-cert\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.831077 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.831024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2hx\" (UniqueName: \"kubernetes.io/projected/93253fe4-60ab-4ac5-9285-f122558b6da2-kube-api-access-2m2hx\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.932140 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.932103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2hx\" (UniqueName: \"kubernetes.io/projected/93253fe4-60ab-4ac5-9285-f122558b6da2-kube-api-access-2m2hx\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.932344 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.932149 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93253fe4-60ab-4ac5-9285-f122558b6da2-webhook-cert\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.934797 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.934775 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93253fe4-60ab-4ac5-9285-f122558b6da2-webhook-cert\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:37.942437 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:37.942415 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2hx\" (UniqueName: \"kubernetes.io/projected/93253fe4-60ab-4ac5-9285-f122558b6da2-kube-api-access-2m2hx\") pod \"maas-controller-5667fb5f9d-clqzj\" (UID: \"93253fe4-60ab-4ac5-9285-f122558b6da2\") " pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:38.036294 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.036204 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a0b5d8-4493-4235-b051-14aa7183933d" path="/var/lib/kubelet/pods/21a0b5d8-4493-4235-b051-14aa7183933d/volumes" Jul 31 18:57:38.082279 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.082246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:38.215464 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.215433 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5667fb5f9d-clqzj"] Jul 31 18:57:38.219093 ip-10-0-139-43 kubenswrapper[2566]: W0731 18:57:38.219064 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93253fe4_60ab_4ac5_9285_f122558b6da2.slice/crio-59d1bb466c1628ecdc4cdd84b304072bbc53e1745891eccbafe924a51e4aacce WatchSource:0}: Error finding container 59d1bb466c1628ecdc4cdd84b304072bbc53e1745891eccbafe924a51e4aacce: Status 404 returned error can't find the container with id 59d1bb466c1628ecdc4cdd84b304072bbc53e1745891eccbafe924a51e4aacce Jul 31 18:57:38.972885 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.972850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" event={"ID":"93253fe4-60ab-4ac5-9285-f122558b6da2","Type":"ContainerStarted","Data":"455c60ecbc10fcbd1f8fceb1cb7957a86a583d257dafc27c25ee9a338e947833"} Jul 31 18:57:38.973241 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.972908 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" event={"ID":"93253fe4-60ab-4ac5-9285-f122558b6da2","Type":"ContainerStarted","Data":"59d1bb466c1628ecdc4cdd84b304072bbc53e1745891eccbafe924a51e4aacce"} Jul 31 18:57:38.973241 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.972945 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 18:57:38.994576 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:38.994526 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" podStartSLOduration=1.589721545 podStartE2EDuration="1.994509349s" totalImagesPullingTime="404.787804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:57:38.994443114 +0000 UTC m=+807.532670763" watchObservedRunningTime="2026-07-31 18:57:38.994509349 +0000 UTC m=+807.532737001" Jul 31 18:57:49.980920 ip-10-0-139-43 kubenswrapper[2566]: I0731 18:57:49.980887 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5667fb5f9d-clqzj" Jul 31 19:00:00.150544 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.150470 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7"] Jul 31 19:00:00.154084 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.154066 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" Jul 31 19:00:00.157151 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.157131 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:00:00.163662 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.163645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" Jul 31 19:00:00.174058 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.174037 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7"] Jul 31 19:00:00.512425 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.512270 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7"] Jul 31 19:00:00.514685 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:00:00.514656 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d77faa5_0282_46d3_9c3d_acf78421ba2b.slice/crio-47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b WatchSource:0}: Error finding container 47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b: Status 404 returned error can't find the container with id 47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b Jul 31 19:00:00.642755 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:00.642723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" event={"ID":"2d77faa5-0282-46d3-9c3d-acf78421ba2b","Type":"ContainerStarted","Data":"47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b"} Jul 31 19:00:04.664373 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:04.664340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" event={"ID":"2d77faa5-0282-46d3-9c3d-acf78421ba2b","Type":"ContainerStarted","Data":"401e84662cb40e580dd011103b121fa829a64056bc64fbc148e2abd63df656b0"} Jul 31 19:00:04.683982 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:04.683927 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" podStartSLOduration=1.269662621 podStartE2EDuration="4.683911365s" totalImagesPullingTime="3.414248744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:00:04.683102878 +0000 UTC m=+953.221330527" watchObservedRunningTime="2026-07-31 19:00:04.683911365 +0000 UTC m=+953.222139016" Jul 31 19:00:05.671203 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:05.671169 2566 generic.go:356] "Generic (PLEG): container finished" podID="2d77faa5-0282-46d3-9c3d-acf78421ba2b" containerID="401e84662cb40e580dd011103b121fa829a64056bc64fbc148e2abd63df656b0" exitCode=0 Jul 31 19:00:05.671662 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:05.671263 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" event={"ID":"2d77faa5-0282-46d3-9c3d-acf78421ba2b","Type":"ContainerDied","Data":"401e84662cb40e580dd011103b121fa829a64056bc64fbc148e2abd63df656b0"} Jul 31 19:00:06.813205 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:06.813181 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" Jul 31 19:00:07.682503 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:07.682460 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" Jul 31 19:00:07.682680 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:07.682518 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758740-tkpg7" event={"ID":"2d77faa5-0282-46d3-9c3d-acf78421ba2b","Type":"ContainerDied","Data":"47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b"} Jul 31 19:00:07.682680 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:00:07.682566 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f259ab55692bf1d803b0e9779d3bd3a8601b2ee22381f68eaa0de20988b10b" Jul 31 19:05:03.003663 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.003584 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:05:03.005095 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.004044 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d77faa5-0282-46d3-9c3d-acf78421ba2b" containerName="cleanup" Jul 31 19:05:03.005095 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.004057 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d77faa5-0282-46d3-9c3d-acf78421ba2b" containerName="cleanup" Jul 31 19:05:03.005095 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.004135 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d77faa5-0282-46d3-9c3d-acf78421ba2b" containerName="cleanup" Jul 31 19:05:03.006446 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.006431 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.011850 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.011803 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 19:05:03.011850 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.011818 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 19:05:03.011850 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.011809 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5kvcw\"" Jul 31 19:05:03.019889 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.019858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:05:03.099146 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.099097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldghk\" (UniqueName: \"kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.099350 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.099169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.200765 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.200721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldghk\" (UniqueName: \"kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.200953 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.200794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.201185 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.201165 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.210818 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.210786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldghk\" (UniqueName: \"kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk\") pod \"kuadrant-operator-controller-manager-68cf589589-nh7bc\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.316450 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.316368 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:03.463030 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:03.462997 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:05:03.465238 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:05:03.465213 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7003a05_9858_460c_a43c_998dad01746e.slice/crio-6f1731dcb42b3c31f0b7dddcbfdf8f4ddafa76c2690ce78e8e0c7a8c50568f70 WatchSource:0}: Error finding container 6f1731dcb42b3c31f0b7dddcbfdf8f4ddafa76c2690ce78e8e0c7a8c50568f70: Status 404 returned error can't find the container with id 6f1731dcb42b3c31f0b7dddcbfdf8f4ddafa76c2690ce78e8e0c7a8c50568f70 Jul 31 19:05:04.123999 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:04.123966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" event={"ID":"e7003a05-9858-460c-a43c-998dad01746e","Type":"ContainerStarted","Data":"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062"} Jul 31 19:05:04.124525 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:04.124022 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" event={"ID":"e7003a05-9858-460c-a43c-998dad01746e","Type":"ContainerStarted","Data":"6f1731dcb42b3c31f0b7dddcbfdf8f4ddafa76c2690ce78e8e0c7a8c50568f70"} Jul 31 19:05:04.124525 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:04.124088 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:05:04.149315 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:04.149255 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" podStartSLOduration=2.149239302 podStartE2EDuration="2.149239302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:05:04.145634435 +0000 UTC m=+1252.683862104" watchObservedRunningTime="2026-07-31 19:05:04.149239302 +0000 UTC m=+1252.687466950" Jul 31 19:05:15.134633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:05:15.134599 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:12:40.848458 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.848377 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:12:40.851647 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.851628 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:40.854430 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.854412 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b30c0866-serving-cert\"" Jul 31 19:12:40.862187 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.862167 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:12:40.953140 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.953105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:40.953309 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.953187 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qt2r\" (UniqueName: \"kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:40.953309 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:40.953216 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.054778 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.054742 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.054947 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.054820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qt2r\" (UniqueName: \"kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.054998 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.054982 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.055674 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.055610 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.057343 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.057320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.064093 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.064071 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qt2r\" (UniqueName: \"kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r\") pod \"maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.162966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.162934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:41.305146 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.305079 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:12:41.307968 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:12:41.307941 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2256afba_f5b7_499e_aedd_5eced36bc5b2.slice/crio-108477b4872f9872fdb6bb35781a4ebdb8b22d8f319b7a46802861f4bd3f5d1e WatchSource:0}: Error finding container 108477b4872f9872fdb6bb35781a4ebdb8b22d8f319b7a46802861f4bd3f5d1e: Status 404 returned error can't find the container with id 108477b4872f9872fdb6bb35781a4ebdb8b22d8f319b7a46802861f4bd3f5d1e Jul 31 19:12:41.309273 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.309253 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:12:41.418313 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.418234 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" event={"ID":"2256afba-f5b7-499e-aedd-5eced36bc5b2","Type":"ContainerStarted","Data":"108477b4872f9872fdb6bb35781a4ebdb8b22d8f319b7a46802861f4bd3f5d1e"} Jul 31 19:12:41.534223 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.534189 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:12:41.537544 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.537516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.540513 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.540478 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b30c0866-dockercfg-4rld4\"" Jul 31 19:12:41.540670 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.540543 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b30c0866\"" Jul 31 19:12:41.547541 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.547519 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:12:41.659973 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.659943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.660156 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.660013 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrc8\" (UniqueName: \"kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.761248 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.761162 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrc8\" (UniqueName: \"kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.761405 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.761249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.761977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.761957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.771665 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.771643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrc8\" (UniqueName: \"kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8\") pod \"payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.851138 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.851094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:41.992322 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:41.991882 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:12:41.997999 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:12:41.997972 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8ddfdd_31b4_4042_b8f1_bb8a466504a3.slice/crio-3ccb0a25a4636b3275118d07e28bed66430619cc7d87869b04fb7b872c9c7d75 WatchSource:0}: Error finding container 3ccb0a25a4636b3275118d07e28bed66430619cc7d87869b04fb7b872c9c7d75: Status 404 returned error can't find the container with id 3ccb0a25a4636b3275118d07e28bed66430619cc7d87869b04fb7b872c9c7d75 Jul 31 19:12:42.425930 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.425895 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" event={"ID":"2256afba-f5b7-499e-aedd-5eced36bc5b2","Type":"ContainerStarted","Data":"739c2cdd25379d2599bccf257c4ed17c38c47243a335e8db044eb9c3eaa4bd9c"} Jul 31 19:12:42.426126 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.426021 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:42.427504 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.427472 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" event={"ID":"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3","Type":"ContainerStarted","Data":"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a"} Jul 31 19:12:42.427633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.427544 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" event={"ID":"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3","Type":"ContainerStarted","Data":"3ccb0a25a4636b3275118d07e28bed66430619cc7d87869b04fb7b872c9c7d75"} Jul 31 19:12:42.427700 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.427638 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:12:42.446164 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.446120 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" podStartSLOduration=1.757547761 podStartE2EDuration="2.446108731s" totalImagesPullingTime="688.56097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:12:42.445185857 +0000 UTC m=+1710.983413509" watchObservedRunningTime="2026-07-31 19:12:42.446108731 +0000 UTC m=+1710.984336381" Jul 31 19:12:42.465796 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:42.465748 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" podStartSLOduration=1.465734108 podStartE2EDuration="1.465734108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:12:42.463122055 +0000 UTC m=+1711.001349705" watchObservedRunningTime="2026-07-31 19:12:42.465734108 +0000 UTC m=+1711.003961758" Jul 31 19:12:48.438464 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:48.438437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:12:53.435226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:12:53.435192 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:13:21.494332 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.494297 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:13:21.494977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.494881 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" podUID="2256afba-f5b7-499e-aedd-5eced36bc5b2" containerName="maas-api" containerID="cri-o://739c2cdd25379d2599bccf257c4ed17c38c47243a335e8db044eb9c3eaa4bd9c" gracePeriod=30 Jul 31 19:13:21.562451 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.562407 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:13:21.562666 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.562633 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" podUID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" containerName="payload-processing" containerID="cri-o://ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a" gracePeriod=30 Jul 31 19:13:21.639579 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.639545 2566 generic.go:356] "Generic (PLEG): container finished" podID="2256afba-f5b7-499e-aedd-5eced36bc5b2" containerID="739c2cdd25379d2599bccf257c4ed17c38c47243a335e8db044eb9c3eaa4bd9c" exitCode=0 Jul 31 19:13:21.639734 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.639616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" event={"ID":"2256afba-f5b7-499e-aedd-5eced36bc5b2","Type":"ContainerDied","Data":"739c2cdd25379d2599bccf257c4ed17c38c47243a335e8db044eb9c3eaa4bd9c"} Jul 31 19:13:21.814828 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.814795 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:13:21.840661 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.840633 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca\") pod \"2256afba-f5b7-499e-aedd-5eced36bc5b2\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " Jul 31 19:13:21.840817 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.840729 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls\") pod \"2256afba-f5b7-499e-aedd-5eced36bc5b2\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " Jul 31 19:13:21.840890 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.840814 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qt2r\" (UniqueName: \"kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r\") pod \"2256afba-f5b7-499e-aedd-5eced36bc5b2\" (UID: \"2256afba-f5b7-499e-aedd-5eced36bc5b2\") " Jul 31 19:13:21.841411 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.841356 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2256afba-f5b7-499e-aedd-5eced36bc5b2" (UID: "2256afba-f5b7-499e-aedd-5eced36bc5b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:13:21.845002 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.844424 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2256afba-f5b7-499e-aedd-5eced36bc5b2" (UID: "2256afba-f5b7-499e-aedd-5eced36bc5b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:13:21.845143 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.845121 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r" (OuterVolumeSpecName: "kube-api-access-7qt2r") pod "2256afba-f5b7-499e-aedd-5eced36bc5b2" (UID: "2256afba-f5b7-499e-aedd-5eced36bc5b2"). InnerVolumeSpecName "kube-api-access-7qt2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:13:21.851437 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.851415 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:13:21.941525 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941453 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume\") pod \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " Jul 31 19:13:21.941706 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941581 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqrc8\" (UniqueName: \"kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8\") pod \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\" (UID: \"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3\") " Jul 31 19:13:21.941884 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941865 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qt2r\" (UniqueName: \"kubernetes.io/projected/2256afba-f5b7-499e-aedd-5eced36bc5b2-kube-api-access-7qt2r\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:21.941922 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941881 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" (UID: "0f8ddfdd-31b4-4042-b8f1-bb8a466504a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:13:21.941922 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941892 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2256afba-f5b7-499e-aedd-5eced36bc5b2-openshift-service-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:21.941991 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.941923 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2256afba-f5b7-499e-aedd-5eced36bc5b2-maas-api-tls\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:21.944094 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:21.944072 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8" (OuterVolumeSpecName: "kube-api-access-tqrc8") pod "0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" (UID: "0f8ddfdd-31b4-4042-b8f1-bb8a466504a3"). InnerVolumeSpecName "kube-api-access-tqrc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:13:22.042445 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.042359 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:22.042445 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.042385 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrc8\" (UniqueName: \"kubernetes.io/projected/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3-kube-api-access-tqrc8\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:22.646214 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.646181 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" Jul 31 19:13:22.646720 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.646217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb" event={"ID":"2256afba-f5b7-499e-aedd-5eced36bc5b2","Type":"ContainerDied","Data":"108477b4872f9872fdb6bb35781a4ebdb8b22d8f319b7a46802861f4bd3f5d1e"} Jul 31 19:13:22.646720 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.646277 2566 scope.go:117] "RemoveContainer" containerID="739c2cdd25379d2599bccf257c4ed17c38c47243a335e8db044eb9c3eaa4bd9c" Jul 31 19:13:22.647762 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.647737 2566 generic.go:356] "Generic (PLEG): container finished" podID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" containerID="ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a" exitCode=0 Jul 31 19:13:22.647860 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.647780 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" Jul 31 19:13:22.647860 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.647788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" event={"ID":"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3","Type":"ContainerDied","Data":"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a"} Jul 31 19:13:22.647860 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.647832 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7" event={"ID":"0f8ddfdd-31b4-4042-b8f1-bb8a466504a3","Type":"ContainerDied","Data":"3ccb0a25a4636b3275118d07e28bed66430619cc7d87869b04fb7b872c9c7d75"} Jul 31 19:13:22.655530 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.655513 2566 scope.go:117] "RemoveContainer" containerID="ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a" Jul 31 19:13:22.663956 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.663931 2566 scope.go:117] "RemoveContainer" containerID="ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a" Jul 31 19:13:22.664245 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:13:22.664225 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a\": container with ID starting with ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a not found: ID does not exist" containerID="ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a" Jul 31 19:13:22.664305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.664254 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a"} err="failed to get container status \"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a\": rpc error: code = NotFound desc = could not find container \"ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a\": container with ID starting with ca2d2fd884ab7da89f60e3116bb94bd1c1c02fe18a3a36ee65e7fa4ec39eb14a not found: ID does not exist" Jul 31 19:13:22.666289 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.666269 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:13:22.668482 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.668460 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b30c0866-66678fd45b-mdbbb"] Jul 31 19:13:22.682388 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.682358 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:13:22.688722 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:22.688697 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b30c0866-6d78d9f5c6-j8qd7"] Jul 31 19:13:24.035669 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:24.035636 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" path="/var/lib/kubelet/pods/0f8ddfdd-31b4-4042-b8f1-bb8a466504a3/volumes" Jul 31 19:13:24.036046 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:24.036018 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2256afba-f5b7-499e-aedd-5eced36bc5b2" path="/var/lib/kubelet/pods/2256afba-f5b7-499e-aedd-5eced36bc5b2/volumes" Jul 31 19:13:25.867597 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.867563 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:25.868002 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.867989 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2256afba-f5b7-499e-aedd-5eced36bc5b2" containerName="maas-api" Jul 31 19:13:25.868064 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.868004 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2256afba-f5b7-499e-aedd-5eced36bc5b2" containerName="maas-api" Jul 31 19:13:25.868064 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.868017 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" containerName="payload-processing" Jul 31 19:13:25.868064 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.868023 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" containerName="payload-processing" Jul 31 19:13:25.868168 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.868090 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2256afba-f5b7-499e-aedd-5eced36bc5b2" containerName="maas-api" Jul 31 19:13:25.868168 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.868103 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8ddfdd-31b4-4042-b8f1-bb8a466504a3" containerName="payload-processing" Jul 31 19:13:25.872759 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.872743 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:25.875982 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.875960 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b9249e59\"" Jul 31 19:13:25.876104 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.875996 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b9249e59-dockercfg-pdvcp\"" Jul 31 19:13:25.882386 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.882360 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:25.975610 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.975578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:25.975610 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:25.975614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thf42\" (UniqueName: \"kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.076980 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.076942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.076980 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.076982 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thf42\" (UniqueName: \"kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.077783 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.077764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.087504 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.087458 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thf42\" (UniqueName: \"kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42\") pod \"payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.184523 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.184475 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.330392 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.330367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:26.332620 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:13:26.332593 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b22211_9c40_4a1d_8a0a_52c3a4a45859.slice/crio-e99a7a0212433f3e7ec649f329e083fd6369d7a0554d08836ad863b56c3d9e07 WatchSource:0}: Error finding container e99a7a0212433f3e7ec649f329e083fd6369d7a0554d08836ad863b56c3d9e07: Status 404 returned error can't find the container with id e99a7a0212433f3e7ec649f329e083fd6369d7a0554d08836ad863b56c3d9e07 Jul 31 19:13:26.671284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.671251 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" event={"ID":"82b22211-9c40-4a1d-8a0a-52c3a4a45859","Type":"ContainerStarted","Data":"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c"} Jul 31 19:13:26.671647 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.671305 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" event={"ID":"82b22211-9c40-4a1d-8a0a-52c3a4a45859","Type":"ContainerStarted","Data":"e99a7a0212433f3e7ec649f329e083fd6369d7a0554d08836ad863b56c3d9e07"} Jul 31 19:13:26.671647 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.671357 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:26.689811 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:26.689769 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" podStartSLOduration=1.689753287 podStartE2EDuration="1.689753287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:13:26.688616353 +0000 UTC m=+1755.226844009" watchObservedRunningTime="2026-07-31 19:13:26.689753287 +0000 UTC m=+1755.227980938" Jul 31 19:13:37.678663 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:37.678629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:56.720342 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:56.720294 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:56.720825 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:56.720777 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" podUID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" containerName="payload-pre-processing" containerID="cri-o://df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c" gracePeriod=30 Jul 31 19:13:56.976810 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:56.976749 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:57.065552 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.065521 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume\") pod \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " Jul 31 19:13:57.065744 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.065590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thf42\" (UniqueName: \"kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42\") pod \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\" (UID: \"82b22211-9c40-4a1d-8a0a-52c3a4a45859\") " Jul 31 19:13:57.065944 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.065919 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82b22211-9c40-4a1d-8a0a-52c3a4a45859" (UID: "82b22211-9c40-4a1d-8a0a-52c3a4a45859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:13:57.067727 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.067706 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42" (OuterVolumeSpecName: "kube-api-access-thf42") pod "82b22211-9c40-4a1d-8a0a-52c3a4a45859" (UID: "82b22211-9c40-4a1d-8a0a-52c3a4a45859"). InnerVolumeSpecName "kube-api-access-thf42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:13:57.166254 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.166218 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b22211-9c40-4a1d-8a0a-52c3a4a45859-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:57.166254 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.166248 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thf42\" (UniqueName: \"kubernetes.io/projected/82b22211-9c40-4a1d-8a0a-52c3a4a45859-kube-api-access-thf42\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:13:57.831726 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.831692 2566 generic.go:356] "Generic (PLEG): container finished" podID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" containerID="df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c" exitCode=0 Jul 31 19:13:57.832305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.831749 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" Jul 31 19:13:57.832305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.831774 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" event={"ID":"82b22211-9c40-4a1d-8a0a-52c3a4a45859","Type":"ContainerDied","Data":"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c"} Jul 31 19:13:57.832305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.831823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9" event={"ID":"82b22211-9c40-4a1d-8a0a-52c3a4a45859","Type":"ContainerDied","Data":"e99a7a0212433f3e7ec649f329e083fd6369d7a0554d08836ad863b56c3d9e07"} Jul 31 19:13:57.832305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.831845 2566 scope.go:117] "RemoveContainer" containerID="df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c" Jul 31 19:13:57.841832 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.841802 2566 scope.go:117] "RemoveContainer" containerID="df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c" Jul 31 19:13:57.842165 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:13:57.842145 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c\": container with ID starting with df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c not found: ID does not exist" containerID="df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c" Jul 31 19:13:57.842240 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.842173 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c"} err="failed to get container status \"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c\": rpc error: code = NotFound desc = could not find container \"df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c\": container with ID starting with df3485d7e408ffd127f1492f9ce5ba3c43ce55b7bfc6192a7fb126aadb709a6c not found: ID does not exist" Jul 31 19:13:57.860746 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.860722 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:57.864579 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:57.864561 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b9249e59-78475b6d4d-qpwl9"] Jul 31 19:13:58.036986 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:13:58.036949 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" path="/var/lib/kubelet/pods/82b22211-9c40-4a1d-8a0a-52c3a4a45859/volumes" Jul 31 19:18:54.700236 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:54.700198 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:18:54.700755 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:54.700447 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" podUID="e7003a05-9858-460c-a43c-998dad01746e" containerName="manager" containerID="cri-o://fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062" gracePeriod=10 Jul 31 19:18:54.949214 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:54.949186 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:18:55.032105 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.032028 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldghk\" (UniqueName: \"kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk\") pod \"e7003a05-9858-460c-a43c-998dad01746e\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " Jul 31 19:18:55.032105 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.032078 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume\") pod \"e7003a05-9858-460c-a43c-998dad01746e\" (UID: \"e7003a05-9858-460c-a43c-998dad01746e\") " Jul 31 19:18:55.032535 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.032512 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7003a05-9858-460c-a43c-998dad01746e" (UID: "e7003a05-9858-460c-a43c-998dad01746e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:18:55.034184 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.034165 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk" (OuterVolumeSpecName: "kube-api-access-ldghk") pod "e7003a05-9858-460c-a43c-998dad01746e" (UID: "e7003a05-9858-460c-a43c-998dad01746e"). InnerVolumeSpecName "kube-api-access-ldghk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:18:55.133656 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.133617 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldghk\" (UniqueName: \"kubernetes.io/projected/e7003a05-9858-460c-a43c-998dad01746e-kube-api-access-ldghk\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:18:55.133656 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.133654 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7003a05-9858-460c-a43c-998dad01746e-extensions-socket-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:18:55.333098 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.333005 2566 generic.go:356] "Generic (PLEG): container finished" podID="e7003a05-9858-460c-a43c-998dad01746e" containerID="fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062" exitCode=0 Jul 31 19:18:55.333098 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.333052 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" Jul 31 19:18:55.333324 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.333095 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" event={"ID":"e7003a05-9858-460c-a43c-998dad01746e","Type":"ContainerDied","Data":"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062"} Jul 31 19:18:55.333324 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.333146 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc" event={"ID":"e7003a05-9858-460c-a43c-998dad01746e","Type":"ContainerDied","Data":"6f1731dcb42b3c31f0b7dddcbfdf8f4ddafa76c2690ce78e8e0c7a8c50568f70"} Jul 31 19:18:55.333324 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.333168 2566 scope.go:117] "RemoveContainer" containerID="fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062" Jul 31 19:18:55.344185 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.344167 2566 scope.go:117] "RemoveContainer" containerID="fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062" Jul 31 19:18:55.344460 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:18:55.344440 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062\": container with ID starting with fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062 not found: ID does not exist" containerID="fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062" Jul 31 19:18:55.344622 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.344469 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062"} err="failed to get container status \"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062\": rpc error: code = NotFound desc = could not find container \"fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062\": container with ID starting with fe51775142e23a30dc50dfbcfa754e174a586ced2dc79b682dfdb1d7a355e062 not found: ID does not exist" Jul 31 19:18:55.359280 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.359250 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:18:55.365463 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:55.365436 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-nh7bc"] Jul 31 19:18:56.034958 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:18:56.034924 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7003a05-9858-460c-a43c-998dad01746e" path="/var/lib/kubelet/pods/e7003a05-9858-460c-a43c-998dad01746e/volumes" Jul 31 19:20:10.319023 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.318943 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv"] Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319355 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7003a05-9858-460c-a43c-998dad01746e" containerName="manager" Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319367 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7003a05-9858-460c-a43c-998dad01746e" containerName="manager" Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319385 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" containerName="payload-pre-processing" Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319390 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" containerName="payload-pre-processing" Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319445 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b22211-9c40-4a1d-8a0a-52c3a4a45859" containerName="payload-pre-processing" Jul 31 19:20:10.319483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.319455 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7003a05-9858-460c-a43c-998dad01746e" containerName="manager" Jul 31 19:20:10.322577 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.322560 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.327336 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.327315 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5kvcw\"" Jul 31 19:20:10.327436 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.327343 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 19:20:10.327436 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.327315 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 19:20:10.344601 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.344569 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv"] Jul 31 19:20:10.441615 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.441576 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96kx\" (UniqueName: \"kubernetes.io/projected/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-kube-api-access-p96kx\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.441615 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.441616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.542267 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.542226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96kx\" (UniqueName: \"kubernetes.io/projected/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-kube-api-access-p96kx\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.542267 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.542270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.542731 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.542712 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.555589 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.555561 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96kx\" (UniqueName: \"kubernetes.io/projected/95a1dbc3-9d82-43df-98b5-5abfba9d5fa7-kube-api-access-p96kx\") pod \"kuadrant-operator-controller-manager-68cf589589-fnnxv\" (UID: \"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.633226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.633137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:10.779028 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:10.778997 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv"] Jul 31 19:20:10.781768 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:20:10.781742 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a1dbc3_9d82_43df_98b5_5abfba9d5fa7.slice/crio-c28008f152751249f5496b9d2234fc975adc7ee5c3514a852d55002c9f45d02d WatchSource:0}: Error finding container c28008f152751249f5496b9d2234fc975adc7ee5c3514a852d55002c9f45d02d: Status 404 returned error can't find the container with id c28008f152751249f5496b9d2234fc975adc7ee5c3514a852d55002c9f45d02d Jul 31 19:20:11.714299 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:11.714257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" event={"ID":"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7","Type":"ContainerStarted","Data":"1c512d779ebe7d0a5d65cd4c111d0ae1e1b28a19fc3f6085776e4ebfc6ddaf77"} Jul 31 19:20:11.714802 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:11.714307 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" event={"ID":"95a1dbc3-9d82-43df-98b5-5abfba9d5fa7","Type":"ContainerStarted","Data":"c28008f152751249f5496b9d2234fc975adc7ee5c3514a852d55002c9f45d02d"} Jul 31 19:20:11.714802 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:11.714389 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:20:11.746583 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:11.746530 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" podStartSLOduration=1.746514758 podStartE2EDuration="1.746514758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:20:11.74496279 +0000 UTC m=+2160.283190445" watchObservedRunningTime="2026-07-31 19:20:11.746514758 +0000 UTC m=+2160.284742399" Jul 31 19:20:22.722356 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:20:22.722316 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-fnnxv" Jul 31 19:24:19.774776 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.774738 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:24:19.778771 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.778744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:19.781913 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.781889 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8aa450\"" Jul 31 19:24:19.782056 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.781980 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8aa450-dockercfg-k4dfv\"" Jul 31 19:24:19.792988 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.792956 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:24:19.926131 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.926090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r65\" (UniqueName: \"kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:19.926131 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:19.926136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.027673 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.027572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r65\" (UniqueName: \"kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.027673 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.027617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.028517 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.028472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.040884 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.040856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r65\" (UniqueName: \"kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65\") pod \"payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.089662 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.089630 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:20.239054 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.239021 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:24:20.241427 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:24:20.241399 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb5ca41_05c1_4211_87a9_175411403a16.slice/crio-349cec7d17a903f859b38678a952976770e0c9eb254d892f842c39e42ca2eb88 WatchSource:0}: Error finding container 349cec7d17a903f859b38678a952976770e0c9eb254d892f842c39e42ca2eb88: Status 404 returned error can't find the container with id 349cec7d17a903f859b38678a952976770e0c9eb254d892f842c39e42ca2eb88 Jul 31 19:24:20.982481 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.982442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" event={"ID":"8eb5ca41-05c1-4211-87a9-175411403a16","Type":"ContainerStarted","Data":"06cc8bd7ed12ca659015b2ca99fc06fae964f7fc0db2e0bda91567460b35741c"} Jul 31 19:24:20.982888 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.982523 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" event={"ID":"8eb5ca41-05c1-4211-87a9-175411403a16","Type":"ContainerStarted","Data":"349cec7d17a903f859b38678a952976770e0c9eb254d892f842c39e42ca2eb88"} Jul 31 19:24:20.982888 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:20.982554 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:24:21.004329 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:21.004276 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" podStartSLOduration=2.004261553 podStartE2EDuration="2.004261553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:24:21.001787743 +0000 UTC m=+2409.540015396" watchObservedRunningTime="2026-07-31 19:24:21.004261553 +0000 UTC m=+2409.542489204" Jul 31 19:24:31.989414 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:24:31.989385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:25:13.793119 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.793088 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:25:13.796629 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.796612 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.801154 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.801130 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8aa450\"/\"default-dockercfg-w796l\"" Jul 31 19:25:13.801273 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.801174 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8aa450\"/\"test-model-8aa450-kserve-self-signed-certs\"" Jul 31 19:25:13.801273 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.801199 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8aa450\"/\"openshift-service-ca.crt\"" Jul 31 19:25:13.801273 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.801238 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8aa450\"/\"kube-root-ca.crt\"" Jul 31 19:25:13.806765 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.806745 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:25:13.853174 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.853369 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.853369 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853253 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nzc\" (UniqueName: \"kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.853369 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853293 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.853571 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.853571 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.853402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954694 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nzc\" (UniqueName: \"kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.954874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.954817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.955133 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.955104 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.955201 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.955181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.955255 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.955229 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.957160 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.957134 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.957415 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.957398 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:13.966889 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:13.966864 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nzc\" (UniqueName: \"kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc\") pod \"test-model-8aa450-kserve-588b494887-xm5k9\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:14.106354 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:14.106271 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:14.250241 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:14.250217 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:25:14.252065 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:25:14.252040 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b964757_c786_4e9e_944b_a4d8987e15ae.slice/crio-4f499dd9f67574ae0b20c12581b6dae8768b51b30798ade379ea3393dc29ac15 WatchSource:0}: Error finding container 4f499dd9f67574ae0b20c12581b6dae8768b51b30798ade379ea3393dc29ac15: Status 404 returned error can't find the container with id 4f499dd9f67574ae0b20c12581b6dae8768b51b30798ade379ea3393dc29ac15 Jul 31 19:25:15.251921 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:15.251887 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" event={"ID":"3b964757-c786-4e9e-944b-a4d8987e15ae","Type":"ContainerStarted","Data":"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea"} Jul 31 19:25:15.252325 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:15.251930 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" event={"ID":"3b964757-c786-4e9e-944b-a4d8987e15ae","Type":"ContainerStarted","Data":"4f499dd9f67574ae0b20c12581b6dae8768b51b30798ade379ea3393dc29ac15"} Jul 31 19:25:15.272766 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:15.272712 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" podStartSLOduration=2.272694559 podStartE2EDuration="2.272694559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:25:15.271873001 +0000 UTC m=+2463.810100699" watchObservedRunningTime="2026-07-31 19:25:15.272694559 +0000 UTC m=+2463.810922210" Jul 31 19:25:24.106736 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:24.106698 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:24.107369 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:24.106882 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:24.119565 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:24.119539 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:24.309636 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:24.309608 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:25:58.084900 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:25:58.084867 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:26:13.105081 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.104995 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" podUID="3b964757-c786-4e9e-944b-a4d8987e15ae" containerName="main" containerID="cri-o://daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea" gracePeriod=45 Jul 31 19:26:13.352239 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.352212 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:26:13.436331 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436298 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4nzc\" (UniqueName: \"kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436353 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436406 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436435 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436470 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436536 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home\") pod \"3b964757-c786-4e9e-944b-a4d8987e15ae\" (UID: \"3b964757-c786-4e9e-944b-a4d8987e15ae\") " Jul 31 19:26:13.436820 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436694 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:13.436820 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436738 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:13.436820 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436806 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home" (OuterVolumeSpecName: "home") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:13.436984 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436916 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-model-cache\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.436984 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436929 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-tmp-dir\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.436984 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.436937 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-home\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.438860 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.438834 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:26:13.438974 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.438878 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm" (OuterVolumeSpecName: "dshm") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:26:13.438974 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.438908 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc" (OuterVolumeSpecName: "kube-api-access-c4nzc") pod "3b964757-c786-4e9e-944b-a4d8987e15ae" (UID: "3b964757-c786-4e9e-944b-a4d8987e15ae"). InnerVolumeSpecName "kube-api-access-c4nzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:13.537481 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.537441 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b964757-c786-4e9e-944b-a4d8987e15ae-tls-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.537481 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.537474 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nzc\" (UniqueName: \"kubernetes.io/projected/3b964757-c786-4e9e-944b-a4d8987e15ae-kube-api-access-c4nzc\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.537722 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.537522 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b964757-c786-4e9e-944b-a4d8987e15ae-dshm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:13.548862 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.548834 2566 generic.go:356] "Generic (PLEG): container finished" podID="3b964757-c786-4e9e-944b-a4d8987e15ae" containerID="daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea" exitCode=0 Jul 31 19:26:13.549013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.548879 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" Jul 31 19:26:13.549013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.548922 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" event={"ID":"3b964757-c786-4e9e-944b-a4d8987e15ae","Type":"ContainerDied","Data":"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea"} Jul 31 19:26:13.549013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.548967 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9" event={"ID":"3b964757-c786-4e9e-944b-a4d8987e15ae","Type":"ContainerDied","Data":"4f499dd9f67574ae0b20c12581b6dae8768b51b30798ade379ea3393dc29ac15"} Jul 31 19:26:13.549013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.548992 2566 scope.go:117] "RemoveContainer" containerID="daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea" Jul 31 19:26:13.558956 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.558939 2566 scope.go:117] "RemoveContainer" containerID="daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea" Jul 31 19:26:13.559223 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:26:13.559204 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea\": container with ID starting with daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea not found: ID does not exist" containerID="daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea" Jul 31 19:26:13.559278 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.559232 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea"} err="failed to get container status \"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea\": rpc error: code = NotFound desc = could not find container \"daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea\": container with ID starting with daf2db17ec45493322714e4a6b24869569b64cede268042f0eb49e4a20d1c9ea not found: ID does not exist" Jul 31 19:26:13.574757 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.574727 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:26:13.579218 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:13.579194 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8aa450/test-model-8aa450-kserve-588b494887-xm5k9"] Jul 31 19:26:14.036430 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:14.036398 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b964757-c786-4e9e-944b-a4d8987e15ae" path="/var/lib/kubelet/pods/3b964757-c786-4e9e-944b-a4d8987e15ae/volumes" Jul 31 19:26:27.452476 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.452438 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:26:27.452952 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.452704 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" podUID="8eb5ca41-05c1-4211-87a9-175411403a16" containerName="payload-processing" containerID="cri-o://06cc8bd7ed12ca659015b2ca99fc06fae964f7fc0db2e0bda91567460b35741c" gracePeriod=30 Jul 31 19:26:27.621904 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.621869 2566 generic.go:356] "Generic (PLEG): container finished" podID="8eb5ca41-05c1-4211-87a9-175411403a16" containerID="06cc8bd7ed12ca659015b2ca99fc06fae964f7fc0db2e0bda91567460b35741c" exitCode=0 Jul 31 19:26:27.622083 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.621947 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" event={"ID":"8eb5ca41-05c1-4211-87a9-175411403a16","Type":"ContainerDied","Data":"06cc8bd7ed12ca659015b2ca99fc06fae964f7fc0db2e0bda91567460b35741c"} Jul 31 19:26:27.720377 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.720353 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:26:27.763633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.763604 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8r65\" (UniqueName: \"kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65\") pod \"8eb5ca41-05c1-4211-87a9-175411403a16\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " Jul 31 19:26:27.763797 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.763639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume\") pod \"8eb5ca41-05c1-4211-87a9-175411403a16\" (UID: \"8eb5ca41-05c1-4211-87a9-175411403a16\") " Jul 31 19:26:27.764131 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.764093 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8eb5ca41-05c1-4211-87a9-175411403a16" (UID: "8eb5ca41-05c1-4211-87a9-175411403a16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:26:27.765758 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.765736 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65" (OuterVolumeSpecName: "kube-api-access-l8r65") pod "8eb5ca41-05c1-4211-87a9-175411403a16" (UID: "8eb5ca41-05c1-4211-87a9-175411403a16"). InnerVolumeSpecName "kube-api-access-l8r65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:26:27.864834 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.864802 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8r65\" (UniqueName: \"kubernetes.io/projected/8eb5ca41-05c1-4211-87a9-175411403a16-kube-api-access-l8r65\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:27.864834 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:27.864829 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5ca41-05c1-4211-87a9-175411403a16-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:26:28.635589 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:28.635558 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" Jul 31 19:26:28.636030 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:28.635558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4" event={"ID":"8eb5ca41-05c1-4211-87a9-175411403a16","Type":"ContainerDied","Data":"349cec7d17a903f859b38678a952976770e0c9eb254d892f842c39e42ca2eb88"} Jul 31 19:26:28.636030 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:28.635727 2566 scope.go:117] "RemoveContainer" containerID="06cc8bd7ed12ca659015b2ca99fc06fae964f7fc0db2e0bda91567460b35741c" Jul 31 19:26:28.656654 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:28.656624 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:26:28.658878 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:28.658855 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8aa450-688c796666-qqwq4"] Jul 31 19:26:30.034976 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:30.034923 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb5ca41-05c1-4211-87a9-175411403a16" path="/var/lib/kubelet/pods/8eb5ca41-05c1-4211-87a9-175411403a16/volumes" Jul 31 19:26:40.503955 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.503907 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:26:40.504344 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504310 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb5ca41-05c1-4211-87a9-175411403a16" containerName="payload-processing" Jul 31 19:26:40.504344 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504321 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb5ca41-05c1-4211-87a9-175411403a16" containerName="payload-processing" Jul 31 19:26:40.504414 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504346 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b964757-c786-4e9e-944b-a4d8987e15ae" containerName="main" Jul 31 19:26:40.504414 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504353 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b964757-c786-4e9e-944b-a4d8987e15ae" containerName="main" Jul 31 19:26:40.504476 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504420 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb5ca41-05c1-4211-87a9-175411403a16" containerName="payload-processing" Jul 31 19:26:40.504476 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.504428 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b964757-c786-4e9e-944b-a4d8987e15ae" containerName="main" Jul 31 19:26:40.508959 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.508940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.512320 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.512301 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-968afa\"" Jul 31 19:26:40.514075 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.514056 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-968afa-dockercfg-b82kb\"" Jul 31 19:26:40.523062 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.523041 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:26:40.581121 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.581078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.581303 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.581213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xzv\" (UniqueName: \"kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.682017 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.681969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.682222 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.682109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xzv\" (UniqueName: \"kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.682787 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.682762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.691854 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.691827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xzv\" (UniqueName: \"kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv\") pod \"payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.819316 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.819221 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:40.960686 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:40.960659 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:26:40.962890 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:26:40.962865 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03866486_c570_407b_9c55_c029d296ddca.slice/crio-4e17b5f0d7afedba3dd589bd0b44bf7fef5d2c74c7592fd604848e30e409c316 WatchSource:0}: Error finding container 4e17b5f0d7afedba3dd589bd0b44bf7fef5d2c74c7592fd604848e30e409c316: Status 404 returned error can't find the container with id 4e17b5f0d7afedba3dd589bd0b44bf7fef5d2c74c7592fd604848e30e409c316 Jul 31 19:26:41.703414 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:41.703378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" event={"ID":"03866486-c570-407b-9c55-c029d296ddca","Type":"ContainerStarted","Data":"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672"} Jul 31 19:26:41.703881 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:41.703433 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" event={"ID":"03866486-c570-407b-9c55-c029d296ddca","Type":"ContainerStarted","Data":"4e17b5f0d7afedba3dd589bd0b44bf7fef5d2c74c7592fd604848e30e409c316"} Jul 31 19:26:41.703881 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:41.703462 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:26:41.723342 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:41.723288 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" podStartSLOduration=1.723273228 podStartE2EDuration="1.723273228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:26:41.722021338 +0000 UTC m=+2550.260248980" watchObservedRunningTime="2026-07-31 19:26:41.723273228 +0000 UTC m=+2550.261500920" Jul 31 19:26:52.711954 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:26:52.711925 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:27:50.247880 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.247772 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:27:50.248694 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.248040 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" podUID="03866486-c570-407b-9c55-c029d296ddca" containerName="payload-pre-processing" containerID="cri-o://9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672" gracePeriod=30 Jul 31 19:27:50.505303 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.505234 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:27:50.538332 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.538300 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume\") pod \"03866486-c570-407b-9c55-c029d296ddca\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " Jul 31 19:27:50.538519 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.538370 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5xzv\" (UniqueName: \"kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv\") pod \"03866486-c570-407b-9c55-c029d296ddca\" (UID: \"03866486-c570-407b-9c55-c029d296ddca\") " Jul 31 19:27:50.538749 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.538723 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03866486-c570-407b-9c55-c029d296ddca" (UID: "03866486-c570-407b-9c55-c029d296ddca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:27:50.540633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.540608 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv" (OuterVolumeSpecName: "kube-api-access-m5xzv") pod "03866486-c570-407b-9c55-c029d296ddca" (UID: "03866486-c570-407b-9c55-c029d296ddca"). InnerVolumeSpecName "kube-api-access-m5xzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:27:50.639737 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.639703 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xzv\" (UniqueName: \"kubernetes.io/projected/03866486-c570-407b-9c55-c029d296ddca-kube-api-access-m5xzv\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:27:50.639737 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:50.639731 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03866486-c570-407b-9c55-c029d296ddca-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:27:51.056515 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.056456 2566 generic.go:356] "Generic (PLEG): container finished" podID="03866486-c570-407b-9c55-c029d296ddca" containerID="9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672" exitCode=0 Jul 31 19:27:51.056701 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.056543 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" event={"ID":"03866486-c570-407b-9c55-c029d296ddca","Type":"ContainerDied","Data":"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672"} Jul 31 19:27:51.056701 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.056566 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" Jul 31 19:27:51.056701 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.056602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4" event={"ID":"03866486-c570-407b-9c55-c029d296ddca","Type":"ContainerDied","Data":"4e17b5f0d7afedba3dd589bd0b44bf7fef5d2c74c7592fd604848e30e409c316"} Jul 31 19:27:51.056701 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.056633 2566 scope.go:117] "RemoveContainer" containerID="9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672" Jul 31 19:27:51.065926 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.065910 2566 scope.go:117] "RemoveContainer" containerID="9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672" Jul 31 19:27:51.066195 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:27:51.066176 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672\": container with ID starting with 9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672 not found: ID does not exist" containerID="9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672" Jul 31 19:27:51.066237 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.066202 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672"} err="failed to get container status \"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672\": rpc error: code = NotFound desc = could not find container \"9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672\": container with ID starting with 9cca910a5bb6cebc66271e40e9ec3fb945562b76f8a3b1dd74742ea13c6cd672 not found: ID does not exist" Jul 31 19:27:51.080112 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.080085 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:27:51.088110 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:51.088073 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968afa-57bb779456-x8qc4"] Jul 31 19:27:52.036593 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:27:52.036561 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03866486-c570-407b-9c55-c029d296ddca" path="/var/lib/kubelet/pods/03866486-c570-407b-9c55-c029d296ddca/volumes" Jul 31 19:29:01.181513 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.180343 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:29:01.181513 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.181340 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03866486-c570-407b-9c55-c029d296ddca" containerName="payload-pre-processing" Jul 31 19:29:01.181513 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.181361 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03866486-c570-407b-9c55-c029d296ddca" containerName="payload-pre-processing" Jul 31 19:29:01.182514 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.182288 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03866486-c570-407b-9c55-c029d296ddca" containerName="payload-pre-processing" Jul 31 19:29:01.186631 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.186603 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.190651 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.190621 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-09a154\"" Jul 31 19:29:01.190795 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.190775 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-09a154-dockercfg-zwm4b\"" Jul 31 19:29:01.196194 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.196169 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:29:01.208052 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.208028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbg5\" (UniqueName: \"kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.208182 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.208103 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.308592 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.308552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbg5\" (UniqueName: \"kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.308783 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.308641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.309256 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.309237 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.333590 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.333554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbg5\" (UniqueName: \"kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5\") pod \"payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.498602 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.498510 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:01.646714 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:01.646676 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:29:01.649038 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:29:01.648986 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655b4fdb_7939_4fbc_9034_9c5995ede111.slice/crio-2cb943ec233cf31814f90996dffce56acea5d76f58380899f733d140a7647da9 WatchSource:0}: Error finding container 2cb943ec233cf31814f90996dffce56acea5d76f58380899f733d140a7647da9: Status 404 returned error can't find the container with id 2cb943ec233cf31814f90996dffce56acea5d76f58380899f733d140a7647da9 Jul 31 19:29:02.416951 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:02.416868 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" event={"ID":"655b4fdb-7939-4fbc-9034-9c5995ede111","Type":"ContainerStarted","Data":"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f"} Jul 31 19:29:02.416951 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:02.416919 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" event={"ID":"655b4fdb-7939-4fbc-9034-9c5995ede111","Type":"ContainerStarted","Data":"2cb943ec233cf31814f90996dffce56acea5d76f58380899f733d140a7647da9"} Jul 31 19:29:02.417443 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:02.417010 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:29:02.436650 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:02.436596 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" podStartSLOduration=1.436580682 podStartE2EDuration="1.436580682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:29:02.434683727 +0000 UTC m=+2690.972911379" watchObservedRunningTime="2026-07-31 19:29:02.436580682 +0000 UTC m=+2690.974808332" Jul 31 19:29:13.424500 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:29:13.424452 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:30:00.139682 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.139646 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f"] Jul 31 19:30:00.143272 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.143255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" Jul 31 19:30:00.146689 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.146665 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:30:00.154187 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.154159 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f"] Jul 31 19:30:00.155589 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.155565 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" Jul 31 19:30:00.293434 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.293406 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f"] Jul 31 19:30:00.294944 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:30:00.294918 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ee67c9_e549_4a00_862b_18a7302db232.slice/crio-83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9 WatchSource:0}: Error finding container 83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9: Status 404 returned error can't find the container with id 83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9 Jul 31 19:30:00.716418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.716379 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" event={"ID":"92ee67c9-e549-4a00-862b-18a7302db232","Type":"ContainerStarted","Data":"51465f8e0a0d5649ff72a0db6dee9d0cee6a104da06c9b8bb5361cb425f9684b"} Jul 31 19:30:00.716633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.716430 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" event={"ID":"92ee67c9-e549-4a00-862b-18a7302db232","Type":"ContainerStarted","Data":"83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9"} Jul 31 19:30:00.735822 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:00.735771 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" podStartSLOduration=0.735756026 podStartE2EDuration="735.756026ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:30:00.73471234 +0000 UTC m=+2749.272939991" watchObservedRunningTime="2026-07-31 19:30:00.735756026 +0000 UTC m=+2749.273983677" Jul 31 19:30:01.722462 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:01.722428 2566 generic.go:356] "Generic (PLEG): container finished" podID="92ee67c9-e549-4a00-862b-18a7302db232" containerID="51465f8e0a0d5649ff72a0db6dee9d0cee6a104da06c9b8bb5361cb425f9684b" exitCode=0 Jul 31 19:30:01.722886 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:01.722517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" event={"ID":"92ee67c9-e549-4a00-862b-18a7302db232","Type":"ContainerDied","Data":"51465f8e0a0d5649ff72a0db6dee9d0cee6a104da06c9b8bb5361cb425f9684b"} Jul 31 19:30:02.871797 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:02.871769 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" Jul 31 19:30:03.733048 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:03.733007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" event={"ID":"92ee67c9-e549-4a00-862b-18a7302db232","Type":"ContainerDied","Data":"83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9"} Jul 31 19:30:03.733048 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:03.733046 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758770-7545f" Jul 31 19:30:03.733261 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:03.733069 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83602eef3c7f716dde48f3ddb011afd2113412788bde9205ff85d77c125727b9" Jul 31 19:30:08.794700 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.794664 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:30:08.795178 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.795100 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ee67c9-e549-4a00-862b-18a7302db232" containerName="cleanup" Jul 31 19:30:08.795178 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.795112 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ee67c9-e549-4a00-862b-18a7302db232" containerName="cleanup" Jul 31 19:30:08.795259 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.795180 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ee67c9-e549-4a00-862b-18a7302db232" containerName="cleanup" Jul 31 19:30:08.798562 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.798539 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.803039 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.803016 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"auth-test-model-09a154-kserve-self-signed-certs\"" Jul 31 19:30:08.803174 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.803064 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"kube-root-ca.crt\"" Jul 31 19:30:08.803174 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.803091 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"openshift-service-ca.crt\"" Jul 31 19:30:08.803174 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.803064 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"default-dockercfg-cqflt\"" Jul 31 19:30:08.812449 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.812422 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:30:08.954037 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.953990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.954226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.954046 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.954226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.954068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.954226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.954115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.954226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.954136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:08.954226 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:08.954156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjf2\" (UniqueName: \"kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.055800 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.055800 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.055800 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjf2\" (UniqueName: \"kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056086 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056086 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056086 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.055921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056259 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.056243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056312 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.056301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.056365 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.056336 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.058304 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.058275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.058709 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.058690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.065524 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.065479 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjf2\" (UniqueName: \"kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2\") pod \"auth-test-model-09a154-kserve-78d778984c-bzds2\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.110353 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.110314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:09.248682 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.248653 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:30:09.250929 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:30:09.250898 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0775e62_ff05_4d89_8859_40c53b3b4eec.slice/crio-fff5b0375ad195bb8e543eac43b59215a347d2944480c099965a8e258e41fbce WatchSource:0}: Error finding container fff5b0375ad195bb8e543eac43b59215a347d2944480c099965a8e258e41fbce: Status 404 returned error can't find the container with id fff5b0375ad195bb8e543eac43b59215a347d2944480c099965a8e258e41fbce Jul 31 19:30:09.765977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.765939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" event={"ID":"d0775e62-ff05-4d89-8859-40c53b3b4eec","Type":"ContainerStarted","Data":"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670"} Jul 31 19:30:09.765977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.765978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" event={"ID":"d0775e62-ff05-4d89-8859-40c53b3b4eec","Type":"ContainerStarted","Data":"fff5b0375ad195bb8e543eac43b59215a347d2944480c099965a8e258e41fbce"} Jul 31 19:30:09.786867 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:09.786815 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" podStartSLOduration=1.786799807 podStartE2EDuration="1.786799807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:30:09.784282343 +0000 UTC m=+2758.322509994" watchObservedRunningTime="2026-07-31 19:30:09.786799807 +0000 UTC m=+2758.325027459" Jul 31 19:30:19.110850 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:19.110810 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:19.111321 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:19.111097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:19.123427 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:19.123405 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:30:19.830335 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:30:19.830305 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:33:52.479030 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:33:52.478988 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:34:07.500905 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.500857 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" podUID="d0775e62-ff05-4d89-8859-40c53b3b4eec" containerName="main" containerID="cri-o://4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670" gracePeriod=45 Jul 31 19:34:07.754393 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.754330 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:34:07.864816 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864779 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864836 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcjf2\" (UniqueName: \"kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864871 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864891 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864924 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.864975 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs\") pod \"d0775e62-ff05-4d89-8859-40c53b3b4eec\" (UID: \"d0775e62-ff05-4d89-8859-40c53b3b4eec\") " Jul 31 19:34:07.865310 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865113 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:07.865310 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865206 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:07.865310 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865213 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home" (OuterVolumeSpecName: "home") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:07.865418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865336 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-model-cache\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.865418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865350 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-home\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.865418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.865359 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-tmp-dir\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.867215 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.867190 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm" (OuterVolumeSpecName: "dshm") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:34:07.867319 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.867260 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:34:07.867319 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.867287 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2" (OuterVolumeSpecName: "kube-api-access-mcjf2") pod "d0775e62-ff05-4d89-8859-40c53b3b4eec" (UID: "d0775e62-ff05-4d89-8859-40c53b3b4eec"). InnerVolumeSpecName "kube-api-access-mcjf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:34:07.966150 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.966114 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0775e62-ff05-4d89-8859-40c53b3b4eec-tls-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.966150 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.966145 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjf2\" (UniqueName: \"kubernetes.io/projected/d0775e62-ff05-4d89-8859-40c53b3b4eec-kube-api-access-mcjf2\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.966150 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.966154 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0775e62-ff05-4d89-8859-40c53b3b4eec-dshm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:34:07.989709 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.989674 2566 generic.go:356] "Generic (PLEG): container finished" podID="d0775e62-ff05-4d89-8859-40c53b3b4eec" containerID="4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670" exitCode=0 Jul 31 19:34:07.989842 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.989718 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" Jul 31 19:34:07.989842 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.989739 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" event={"ID":"d0775e62-ff05-4d89-8859-40c53b3b4eec","Type":"ContainerDied","Data":"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670"} Jul 31 19:34:07.989842 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.989777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2" event={"ID":"d0775e62-ff05-4d89-8859-40c53b3b4eec","Type":"ContainerDied","Data":"fff5b0375ad195bb8e543eac43b59215a347d2944480c099965a8e258e41fbce"} Jul 31 19:34:07.989842 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.989799 2566 scope.go:117] "RemoveContainer" containerID="4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670" Jul 31 19:34:07.999004 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.998983 2566 scope.go:117] "RemoveContainer" containerID="4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670" Jul 31 19:34:07.999429 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:34:07.999295 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670\": container with ID starting with 4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670 not found: ID does not exist" containerID="4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670" Jul 31 19:34:07.999429 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:07.999329 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670"} err="failed to get container status \"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670\": rpc error: code = NotFound desc = could not find container \"4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670\": container with ID starting with 4442c09b1c524540c7d0f7039a796e63515fb647acca60e2ff580cc1b7c42670 not found: ID does not exist" Jul 31 19:34:08.015861 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:08.015792 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:34:08.020216 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:08.020188 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/auth-test-model-09a154-kserve-78d778984c-bzds2"] Jul 31 19:34:08.035180 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:08.035150 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0775e62-ff05-4d89-8859-40c53b3b4eec" path="/var/lib/kubelet/pods/d0775e62-ff05-4d89-8859-40c53b3b4eec/volumes" Jul 31 19:34:25.281265 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.281228 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:34:25.281793 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.281673 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0775e62-ff05-4d89-8859-40c53b3b4eec" containerName="main" Jul 31 19:34:25.281793 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.281685 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0775e62-ff05-4d89-8859-40c53b3b4eec" containerName="main" Jul 31 19:34:25.281793 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.281763 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0775e62-ff05-4d89-8859-40c53b3b4eec" containerName="main" Jul 31 19:34:25.284947 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.284927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.287987 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.287964 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"openshift-service-ca.crt\"" Jul 31 19:34:25.289449 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.289424 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"kube-root-ca.crt\"" Jul 31 19:34:25.289624 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.289424 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"sub-test-model-09a154-kserve-self-signed-certs\"" Jul 31 19:34:25.289624 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.289512 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"default-dockercfg-cqflt\"" Jul 31 19:34:25.296638 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.296610 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:34:25.339636 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.339636 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.339874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc7t\" (UniqueName: \"kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.339874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.339874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339785 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.339874 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.339823 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440769 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440859 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc7t\" (UniqueName: \"kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440892 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.440966 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.440942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.441235 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.441169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.441235 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.441216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.441320 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.441262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.443502 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.443463 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.443652 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.443633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.449876 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.449852 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc7t\" (UniqueName: \"kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t\") pod \"sub-test-model-09a154-kserve-5d889cddbc-ng6vt\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.597757 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.597650 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:25.738347 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:25.738322 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:34:25.739994 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:34:25.739963 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc6505c_f998_42ea_bf96_a4d86f22a89a.slice/crio-8209e034ce1ec9f0ef227d33715882dbc8ddb7d99386c499f1aadc23e99a5a0c WatchSource:0}: Error finding container 8209e034ce1ec9f0ef227d33715882dbc8ddb7d99386c499f1aadc23e99a5a0c: Status 404 returned error can't find the container with id 8209e034ce1ec9f0ef227d33715882dbc8ddb7d99386c499f1aadc23e99a5a0c Jul 31 19:34:26.090443 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:26.090406 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" event={"ID":"6bc6505c-f998-42ea-bf96-a4d86f22a89a","Type":"ContainerStarted","Data":"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9"} Jul 31 19:34:26.090443 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:26.090448 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" event={"ID":"6bc6505c-f998-42ea-bf96-a4d86f22a89a","Type":"ContainerStarted","Data":"8209e034ce1ec9f0ef227d33715882dbc8ddb7d99386c499f1aadc23e99a5a0c"} Jul 31 19:34:26.110386 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:26.110337 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" podStartSLOduration=1.110321919 podStartE2EDuration="1.110321919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:34:26.109104988 +0000 UTC m=+3014.647332641" watchObservedRunningTime="2026-07-31 19:34:26.110321919 +0000 UTC m=+3014.648549569" Jul 31 19:34:35.598584 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:35.598548 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:35.599078 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:35.598600 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:35.611133 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:35.611071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:34:36.152649 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:34:36.152616 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:35:48.982074 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:35:48.982033 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:36:04.002467 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.002399 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" podUID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" containerName="main" containerID="cri-o://1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9" gracePeriod=45 Jul 31 19:36:04.254428 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.254367 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:36:04.314460 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314432 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314466 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314509 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314544 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkc7t\" (UniqueName: \"kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314598 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314646 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs\") pod \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\" (UID: \"6bc6505c-f998-42ea-bf96-a4d86f22a89a\") " Jul 31 19:36:04.314907 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314766 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home" (OuterVolumeSpecName: "home") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:04.314907 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314778 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:04.314907 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.314852 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:04.315068 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.315048 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-model-cache\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.315118 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.315073 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-home\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.315118 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.315088 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tmp-dir\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.316788 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.316765 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t" (OuterVolumeSpecName: "kube-api-access-pkc7t") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "kube-api-access-pkc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:36:04.316852 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.316827 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm" (OuterVolumeSpecName: "dshm") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:36:04.316888 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.316857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bc6505c-f998-42ea-bf96-a4d86f22a89a" (UID: "6bc6505c-f998-42ea-bf96-a4d86f22a89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:36:04.416020 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.415984 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc6505c-f998-42ea-bf96-a4d86f22a89a-tls-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.416020 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.416010 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc6505c-f998-42ea-bf96-a4d86f22a89a-dshm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.416020 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.416019 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkc7t\" (UniqueName: \"kubernetes.io/projected/6bc6505c-f998-42ea-bf96-a4d86f22a89a-kube-api-access-pkc7t\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:36:04.603343 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.603258 2566 generic.go:356] "Generic (PLEG): container finished" podID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" containerID="1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9" exitCode=0 Jul 31 19:36:04.603343 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.603301 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" Jul 31 19:36:04.603554 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.603339 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" event={"ID":"6bc6505c-f998-42ea-bf96-a4d86f22a89a","Type":"ContainerDied","Data":"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9"} Jul 31 19:36:04.603554 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.603379 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt" event={"ID":"6bc6505c-f998-42ea-bf96-a4d86f22a89a","Type":"ContainerDied","Data":"8209e034ce1ec9f0ef227d33715882dbc8ddb7d99386c499f1aadc23e99a5a0c"} Jul 31 19:36:04.603554 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.603396 2566 scope.go:117] "RemoveContainer" containerID="1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9" Jul 31 19:36:04.612687 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.612667 2566 scope.go:117] "RemoveContainer" containerID="1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9" Jul 31 19:36:04.612916 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:36:04.612900 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9\": container with ID starting with 1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9 not found: ID does not exist" containerID="1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9" Jul 31 19:36:04.612977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.612922 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9"} err="failed to get container status \"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9\": rpc error: code = NotFound desc = could not find container \"1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9\": container with ID starting with 1dc27849db62ff0f426299882f804794162f501b5a399191fe38b33832909dd9 not found: ID does not exist" Jul 31 19:36:04.626702 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.626679 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:36:04.632195 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:04.632176 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/sub-test-model-09a154-kserve-5d889cddbc-ng6vt"] Jul 31 19:36:06.035800 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:06.035765 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" path="/var/lib/kubelet/pods/6bc6505c-f998-42ea-bf96-a4d86f22a89a/volumes" Jul 31 19:36:27.294455 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.294417 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:36:27.294956 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.294890 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" containerName="main" Jul 31 19:36:27.294956 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.294902 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" containerName="main" Jul 31 19:36:27.295033 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.294978 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc6505c-f998-42ea-bf96-a4d86f22a89a" containerName="main" Jul 31 19:36:27.298029 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.298013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.303117 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.303088 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"kube-root-ca.crt\"" Jul 31 19:36:27.303117 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.303095 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"rate-test-model-09a154-kserve-self-signed-certs\"" Jul 31 19:36:27.303295 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.303108 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"openshift-service-ca.crt\"" Jul 31 19:36:27.303295 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.303157 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-09a154\"/\"default-dockercfg-cqflt\"" Jul 31 19:36:27.309189 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.309164 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:36:27.441444 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.441665 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.441665 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.441665 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dnm\" (UniqueName: \"kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.441831 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.441831 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.441722 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543179 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543372 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543372 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543372 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543372 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543277 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dnm\" (UniqueName: \"kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543372 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543725 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543692 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543788 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543750 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.543835 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.543792 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.545755 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.545689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.546023 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.546001 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.555768 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.555744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dnm\" (UniqueName: \"kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm\") pod \"rate-test-model-09a154-kserve-75dff9dd95-h2gk9\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.607184 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.607150 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:27.755153 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:27.755123 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:36:27.757641 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:36:27.757613 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a1deb6e_c6bb_4b8a_8d9d_0488a6e8c45d.slice/crio-d1f19f0f909a2ae68c7a0b6565170f5bc2c4b7b3deda69e7dd060f760335a6ab WatchSource:0}: Error finding container d1f19f0f909a2ae68c7a0b6565170f5bc2c4b7b3deda69e7dd060f760335a6ab: Status 404 returned error can't find the container with id d1f19f0f909a2ae68c7a0b6565170f5bc2c4b7b3deda69e7dd060f760335a6ab Jul 31 19:36:28.727704 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:28.727668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" event={"ID":"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d","Type":"ContainerStarted","Data":"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5"} Jul 31 19:36:28.728092 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:28.727717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" event={"ID":"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d","Type":"ContainerStarted","Data":"d1f19f0f909a2ae68c7a0b6565170f5bc2c4b7b3deda69e7dd060f760335a6ab"} Jul 31 19:36:28.749858 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:28.749805 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" podStartSLOduration=1.7497895620000001 podStartE2EDuration="1.749789562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:36:28.748150785 +0000 UTC m=+3137.286378437" watchObservedRunningTime="2026-07-31 19:36:28.749789562 +0000 UTC m=+3137.288017213" Jul 31 19:36:37.608036 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:37.607949 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:37.608455 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:37.608138 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:37.620860 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:37.620838 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:36:37.783902 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:36:37.783870 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:38:07.467383 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:07.467305 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:38:08.441333 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.441301 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266710064"] Jul 31 19:38:08.445003 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.444981 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:08.447832 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.447813 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4rlcb\"" Jul 31 19:38:08.452212 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.452189 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266710064"] Jul 31 19:38:08.504114 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.504079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94522\" (UniqueName: \"kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522\") pod \"test-curl-7593-139919266710064\" (UID: \"965f1fad-be25-4647-86a5-96d059db943c\") " pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:08.605384 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.605342 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94522\" (UniqueName: \"kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522\") pod \"test-curl-7593-139919266710064\" (UID: \"965f1fad-be25-4647-86a5-96d059db943c\") " pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:08.615508 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.615471 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94522\" (UniqueName: \"kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522\") pod \"test-curl-7593-139919266710064\" (UID: \"965f1fad-be25-4647-86a5-96d059db943c\") " pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:08.755880 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.755791 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:08.899029 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.899002 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266710064"] Jul 31 19:38:08.901681 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:38:08.901651 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod965f1fad_be25_4647_86a5_96d059db943c.slice/crio-c2d34eb264b3fc1f2752b3a41f44dc896f2695b2bd9944f1fa6bf69e2e7b422a WatchSource:0}: Error finding container c2d34eb264b3fc1f2752b3a41f44dc896f2695b2bd9944f1fa6bf69e2e7b422a: Status 404 returned error can't find the container with id c2d34eb264b3fc1f2752b3a41f44dc896f2695b2bd9944f1fa6bf69e2e7b422a Jul 31 19:38:08.903054 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:08.903032 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 19:38:09.249428 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:09.249387 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266710064" event={"ID":"965f1fad-be25-4647-86a5-96d059db943c","Type":"ContainerStarted","Data":"c2d34eb264b3fc1f2752b3a41f44dc896f2695b2bd9944f1fa6bf69e2e7b422a"} Jul 31 19:38:10.256374 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:10.256338 2566 generic.go:356] "Generic (PLEG): container finished" podID="965f1fad-be25-4647-86a5-96d059db943c" containerID="7f557b4a0ed9243848b9da7796973d2f650b07238b5ead78f8c5c67286b9cc3c" exitCode=0 Jul 31 19:38:10.256822 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:10.256399 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266710064" event={"ID":"965f1fad-be25-4647-86a5-96d059db943c","Type":"ContainerDied","Data":"7f557b4a0ed9243848b9da7796973d2f650b07238b5ead78f8c5c67286b9cc3c"} Jul 31 19:38:11.400380 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.400352 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:11.437849 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.437825 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266710064_965f1fad-be25-4647-86a5-96d059db943c/test-curl-7593-139919266710064/0.log" Jul 31 19:38:11.463804 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.463769 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266710064"] Jul 31 19:38:11.467647 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.467620 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266710064"] Jul 31 19:38:11.535143 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.535053 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94522\" (UniqueName: \"kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522\") pod \"965f1fad-be25-4647-86a5-96d059db943c\" (UID: \"965f1fad-be25-4647-86a5-96d059db943c\") " Jul 31 19:38:11.537281 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.537256 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522" (OuterVolumeSpecName: "kube-api-access-94522") pod "965f1fad-be25-4647-86a5-96d059db943c" (UID: "965f1fad-be25-4647-86a5-96d059db943c"). InnerVolumeSpecName "kube-api-access-94522". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:11.585076 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.585040 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266712880"] Jul 31 19:38:11.585557 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.585541 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965f1fad-be25-4647-86a5-96d059db943c" containerName="test-curl-7593-139919266710064" Jul 31 19:38:11.585608 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.585559 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="965f1fad-be25-4647-86a5-96d059db943c" containerName="test-curl-7593-139919266710064" Jul 31 19:38:11.585657 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.585647 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="965f1fad-be25-4647-86a5-96d059db943c" containerName="test-curl-7593-139919266710064" Jul 31 19:38:11.588885 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.588869 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:11.593774 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.593745 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266712880"] Jul 31 19:38:11.636705 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.636667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkgr\" (UniqueName: \"kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr\") pod \"test-curl-7593-139919266712880\" (UID: \"6363a8a3-48b5-45d8-8bca-667d2a966e6f\") " pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:11.636883 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.636831 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94522\" (UniqueName: \"kubernetes.io/projected/965f1fad-be25-4647-86a5-96d059db943c-kube-api-access-94522\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:11.737893 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.737849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkgr\" (UniqueName: \"kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr\") pod \"test-curl-7593-139919266712880\" (UID: \"6363a8a3-48b5-45d8-8bca-667d2a966e6f\") " pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:11.747695 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.747661 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkgr\" (UniqueName: \"kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr\") pod \"test-curl-7593-139919266712880\" (UID: \"6363a8a3-48b5-45d8-8bca-667d2a966e6f\") " pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:11.900904 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:11.900871 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:12.037790 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:12.037747 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965f1fad-be25-4647-86a5-96d059db943c" path="/var/lib/kubelet/pods/965f1fad-be25-4647-86a5-96d059db943c/volumes" Jul 31 19:38:12.040173 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:12.040146 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266712880"] Jul 31 19:38:12.042453 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:38:12.042430 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6363a8a3_48b5_45d8_8bca_667d2a966e6f.slice/crio-c8c64cbaa952fd33fc67bfaf01f0bb1d21699e23bd459d334616aec694b7fe07 WatchSource:0}: Error finding container c8c64cbaa952fd33fc67bfaf01f0bb1d21699e23bd459d334616aec694b7fe07: Status 404 returned error can't find the container with id c8c64cbaa952fd33fc67bfaf01f0bb1d21699e23bd459d334616aec694b7fe07 Jul 31 19:38:12.267906 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:12.267812 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266712880" event={"ID":"6363a8a3-48b5-45d8-8bca-667d2a966e6f","Type":"ContainerStarted","Data":"c8c64cbaa952fd33fc67bfaf01f0bb1d21699e23bd459d334616aec694b7fe07"} Jul 31 19:38:12.268948 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:12.268926 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266710064" Jul 31 19:38:12.268948 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:12.268939 2566 scope.go:117] "RemoveContainer" containerID="7f557b4a0ed9243848b9da7796973d2f650b07238b5ead78f8c5c67286b9cc3c" Jul 31 19:38:13.276037 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:13.276006 2566 generic.go:356] "Generic (PLEG): container finished" podID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" containerID="40a779d7f5c80f3a9aff0139383dcc17ca07782f77106a639a53010e9f16f529" exitCode=0 Jul 31 19:38:13.276444 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:13.276066 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266712880" event={"ID":"6363a8a3-48b5-45d8-8bca-667d2a966e6f","Type":"ContainerDied","Data":"40a779d7f5c80f3a9aff0139383dcc17ca07782f77106a639a53010e9f16f529"} Jul 31 19:38:14.424449 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.424422 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:14.445392 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.445361 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266712880_6363a8a3-48b5-45d8-8bca-667d2a966e6f/test-curl-7593-139919266712880/0.log" Jul 31 19:38:14.467480 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.467444 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266712880"] Jul 31 19:38:14.471305 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.471279 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266712880"] Jul 31 19:38:14.565481 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.565387 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtkgr\" (UniqueName: \"kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr\") pod \"6363a8a3-48b5-45d8-8bca-667d2a966e6f\" (UID: \"6363a8a3-48b5-45d8-8bca-667d2a966e6f\") " Jul 31 19:38:14.567633 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.567608 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr" (OuterVolumeSpecName: "kube-api-access-wtkgr") pod "6363a8a3-48b5-45d8-8bca-667d2a966e6f" (UID: "6363a8a3-48b5-45d8-8bca-667d2a966e6f"). InnerVolumeSpecName "kube-api-access-wtkgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:14.666861 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:14.666825 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkgr\" (UniqueName: \"kubernetes.io/projected/6363a8a3-48b5-45d8-8bca-667d2a966e6f-kube-api-access-wtkgr\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:15.289577 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:15.289546 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c64cbaa952fd33fc67bfaf01f0bb1d21699e23bd459d334616aec694b7fe07" Jul 31 19:38:15.289754 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:15.289613 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:15.299992 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:38:15.299963 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7593-139919266712880\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" pod="opendatahub/test-curl-7593-139919266712880" Jul 31 19:38:16.035010 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:16.034976 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" path="/var/lib/kubelet/pods/6363a8a3-48b5-45d8-8bca-667d2a966e6f/volumes" Jul 31 19:38:17.644747 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.644714 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:17.645128 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.645115 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" containerName="test-curl-7593-139919266712880" Jul 31 19:38:17.645181 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.645129 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" containerName="test-curl-7593-139919266712880" Jul 31 19:38:17.645230 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.645221 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6363a8a3-48b5-45d8-8bca-667d2a966e6f" containerName="test-curl-7593-139919266712880" Jul 31 19:38:17.648566 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.648551 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:17.651398 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.651367 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4rlcb\"" Jul 31 19:38:17.654977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.654957 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:17.799189 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.799147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwftr\" (UniqueName: \"kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr\") pod \"test-curl-7593-139919266618512\" (UID: \"de3e9910-b446-456d-9532-ce66a4c8f891\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:17.899777 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.899701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwftr\" (UniqueName: \"kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr\") pod \"test-curl-7593-139919266618512\" (UID: \"de3e9910-b446-456d-9532-ce66a4c8f891\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:17.909541 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.909519 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwftr\" (UniqueName: \"kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr\") pod \"test-curl-7593-139919266618512\" (UID: \"de3e9910-b446-456d-9532-ce66a4c8f891\") " pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:17.958281 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:17.958246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:18.095603 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:18.095572 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:18.097514 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:38:18.097464 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3e9910_b446_456d_9532_ce66a4c8f891.slice/crio-04bb555108fd25ff2d92086bb95af8a1a0b4a04938654fc42a073e72cb1e7072 WatchSource:0}: Error finding container 04bb555108fd25ff2d92086bb95af8a1a0b4a04938654fc42a073e72cb1e7072: Status 404 returned error can't find the container with id 04bb555108fd25ff2d92086bb95af8a1a0b4a04938654fc42a073e72cb1e7072 Jul 31 19:38:18.307609 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:18.307515 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266618512" event={"ID":"de3e9910-b446-456d-9532-ce66a4c8f891","Type":"ContainerStarted","Data":"04bb555108fd25ff2d92086bb95af8a1a0b4a04938654fc42a073e72cb1e7072"} Jul 31 19:38:19.315679 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:19.315585 2566 generic.go:356] "Generic (PLEG): container finished" podID="de3e9910-b446-456d-9532-ce66a4c8f891" containerID="253acdac35b6b6cb5e4a321b2c1fa337b9053a77900254b08129780922927b7f" exitCode=0 Jul 31 19:38:19.315679 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:19.315657 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-139919266618512" event={"ID":"de3e9910-b446-456d-9532-ce66a4c8f891","Type":"ContainerDied","Data":"253acdac35b6b6cb5e4a321b2c1fa337b9053a77900254b08129780922927b7f"} Jul 31 19:38:20.459429 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.459405 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:20.481612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.481588 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7593-139919266618512_de3e9910-b446-456d-9532-ce66a4c8f891/test-curl-7593-139919266618512/0.log" Jul 31 19:38:20.504178 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.504152 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:20.508224 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.508199 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-139919266618512"] Jul 31 19:38:20.623540 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.623413 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwftr\" (UniqueName: \"kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr\") pod \"de3e9910-b446-456d-9532-ce66a4c8f891\" (UID: \"de3e9910-b446-456d-9532-ce66a4c8f891\") " Jul 31 19:38:20.625715 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.625690 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr" (OuterVolumeSpecName: "kube-api-access-pwftr") pod "de3e9910-b446-456d-9532-ce66a4c8f891" (UID: "de3e9910-b446-456d-9532-ce66a4c8f891"). InnerVolumeSpecName "kube-api-access-pwftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:20.724601 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:20.724550 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwftr\" (UniqueName: \"kubernetes.io/projected/de3e9910-b446-456d-9532-ce66a4c8f891-kube-api-access-pwftr\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:21.327501 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:21.327470 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:21.327677 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:21.327480 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04bb555108fd25ff2d92086bb95af8a1a0b4a04938654fc42a073e72cb1e7072" Jul 31 19:38:21.337479 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:38:21.337449 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7593-139919266618512\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:22.034869 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:38:22.034834 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7593-139919266618512\" is forbidden: User \"system:node:ip-10-0-139-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-43.ec2.internal' and this object" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" pod="opendatahub/test-curl-7593-139919266618512" Jul 31 19:38:22.035745 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.035726 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" path="/var/lib/kubelet/pods/de3e9910-b446-456d-9532-ce66a4c8f891/volumes" Jul 31 19:38:22.487279 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.487233 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" podUID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" containerName="main" containerID="cri-o://b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5" gracePeriod=45 Jul 31 19:38:22.752047 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.751984 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:38:22.845101 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845061 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845101 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845103 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845366 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845155 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845366 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845184 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845366 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845203 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845366 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845227 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22dnm\" (UniqueName: \"kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm\") pod \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\" (UID: \"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d\") " Jul 31 19:38:22.845612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845391 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home" (OuterVolumeSpecName: "home") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.845612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845524 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.845612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845538 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.845612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.845576 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-home\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.847543 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.847521 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:38:22.847630 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.847553 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm" (OuterVolumeSpecName: "kube-api-access-22dnm") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "kube-api-access-22dnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:38:22.847630 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.847618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm" (OuterVolumeSpecName: "dshm") pod "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" (UID: "7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:38:22.946853 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.946810 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tmp-dir\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.946853 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.946846 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-model-cache\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.946853 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.946857 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dnm\" (UniqueName: \"kubernetes.io/projected/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-kube-api-access-22dnm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.946853 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.946867 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-dshm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:22.947128 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:22.946875 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d-tls-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:38:23.340047 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.340016 2566 generic.go:356] "Generic (PLEG): container finished" podID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" containerID="b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5" exitCode=0 Jul 31 19:38:23.340567 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.340094 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" Jul 31 19:38:23.340567 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.340097 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" event={"ID":"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d","Type":"ContainerDied","Data":"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5"} Jul 31 19:38:23.340567 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.340208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9" event={"ID":"7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d","Type":"ContainerDied","Data":"d1f19f0f909a2ae68c7a0b6565170f5bc2c4b7b3deda69e7dd060f760335a6ab"} Jul 31 19:38:23.340567 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.340226 2566 scope.go:117] "RemoveContainer" containerID="b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5" Jul 31 19:38:23.349364 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.349344 2566 scope.go:117] "RemoveContainer" containerID="b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5" Jul 31 19:38:23.349644 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:38:23.349623 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5\": container with ID starting with b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5 not found: ID does not exist" containerID="b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5" Jul 31 19:38:23.349712 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.349653 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5"} err="failed to get container status \"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5\": rpc error: code = NotFound desc = could not find container \"b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5\": container with ID starting with b7114f108d5218fd78629f303cc282a6bbc6dde135ca27221c1d98535cb9b0d5 not found: ID does not exist" Jul 31 19:38:23.363034 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.363007 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:38:23.367143 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:23.367120 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-09a154/rate-test-model-09a154-kserve-75dff9dd95-h2gk9"] Jul 31 19:38:24.035461 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:24.035428 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" path="/var/lib/kubelet/pods/7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d/volumes" Jul 31 19:38:32.287014 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.286979 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:38:32.287410 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287398 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" containerName="main" Jul 31 19:38:32.287465 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287411 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" containerName="main" Jul 31 19:38:32.287465 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287419 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" containerName="test-curl-7593-139919266618512" Jul 31 19:38:32.287465 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287425 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" containerName="test-curl-7593-139919266618512" Jul 31 19:38:32.287599 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287507 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e9910-b446-456d-9532-ce66a4c8f891" containerName="test-curl-7593-139919266618512" Jul 31 19:38:32.287599 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.287518 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1deb6e-c6bb-4b8a-8d9d-0488a6e8c45d" containerName="main" Jul 31 19:38:32.291884 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.291858 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.295382 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.295360 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0368ff\"" Jul 31 19:38:32.302922 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.302901 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:38:32.315866 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.315842 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:38:32.319599 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.319579 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.323374 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.323352 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0368ff-dockercfg-nmdtf\"" Jul 31 19:38:32.332412 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.332389 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:38:32.436053 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.436014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvvx\" (UniqueName: \"kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.436250 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.436074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.436250 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.436176 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6nn\" (UniqueName: \"kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.436250 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.436227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.537404 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.537304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6nn\" (UniqueName: \"kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.537404 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.537361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.537684 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.537449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvvx\" (UniqueName: \"kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.537684 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.537539 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.538131 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.538107 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.538256 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.538237 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.546780 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.546755 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvvx\" (UniqueName: \"kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx\") pod \"payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.546891 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.546869 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6nn\" (UniqueName: \"kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn\") pod \"payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.603291 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.603247 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:32.631632 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.631463 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:32.796931 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.796897 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:38:32.799222 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:38:32.799195 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d1154_d0c8_40f0_b2ee_f0fe9d70f907.slice/crio-550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad WatchSource:0}: Error finding container 550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad: Status 404 returned error can't find the container with id 550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad Jul 31 19:38:32.972352 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:32.972327 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:38:32.973951 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:38:32.973926 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e90b15_c3e0_4ddb_bee2_f56feb5a0770.slice/crio-9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124 WatchSource:0}: Error finding container 9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124: Status 404 returned error can't find the container with id 9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124 Jul 31 19:38:33.393858 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.393809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" event={"ID":"235d1154-d0c8-40f0-b2ee-f0fe9d70f907","Type":"ContainerStarted","Data":"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71"} Jul 31 19:38:33.394348 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.393889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" event={"ID":"235d1154-d0c8-40f0-b2ee-f0fe9d70f907","Type":"ContainerStarted","Data":"550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad"} Jul 31 19:38:33.394348 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.393934 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:33.396263 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.396232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" event={"ID":"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770","Type":"ContainerStarted","Data":"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb"} Jul 31 19:38:33.396405 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.396289 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" event={"ID":"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770","Type":"ContainerStarted","Data":"9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124"} Jul 31 19:38:33.396474 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.396405 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:38:33.415036 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.414982 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" podStartSLOduration=1.414966168 podStartE2EDuration="1.414966168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:38:33.412811642 +0000 UTC m=+3261.951039299" watchObservedRunningTime="2026-07-31 19:38:33.414966168 +0000 UTC m=+3261.953193818" Jul 31 19:38:33.432715 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:33.432660 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" podStartSLOduration=1.432645935 podStartE2EDuration="1.432645935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:38:33.430888127 +0000 UTC m=+3261.969115778" watchObservedRunningTime="2026-07-31 19:38:33.432645935 +0000 UTC m=+3261.970873585" Jul 31 19:38:44.409109 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:44.409073 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:38:44.409740 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:38:44.409251 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:39:06.314069 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.314024 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:39:06.317516 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.317476 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.320259 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.320235 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90f898\"" Jul 31 19:39:06.336351 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.336320 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:39:06.363964 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.363929 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:39:06.368403 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.368380 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.371539 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.371519 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90f898-dockercfg-6wnsd\"" Jul 31 19:39:06.377923 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.377893 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:39:06.463769 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.463732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmwc\" (UniqueName: \"kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.463928 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.463804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.463928 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.463838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.463928 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.463906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnjj\" (UniqueName: \"kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.564759 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.564667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmwc\" (UniqueName: \"kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.564759 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.564709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.564759 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.564729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.565043 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.564867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnjj\" (UniqueName: \"kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.565380 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.565360 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.565483 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.565466 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.573588 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.573566 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmwc\" (UniqueName: \"kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc\") pod \"payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.573713 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.573604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnjj\" (UniqueName: \"kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj\") pod \"payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.635102 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.635071 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:06.680078 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.680048 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:06.791387 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.791359 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:39:06.794424 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:39:06.794377 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb7c412_bb2b_4d6a_9cf7_edf0b31ab558.slice/crio-18460d80035f338735aa057e21416626d7c673a514f1a1207cd0e738fb3b7715 WatchSource:0}: Error finding container 18460d80035f338735aa057e21416626d7c673a514f1a1207cd0e738fb3b7715: Status 404 returned error can't find the container with id 18460d80035f338735aa057e21416626d7c673a514f1a1207cd0e738fb3b7715 Jul 31 19:39:06.853390 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:06.853367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:39:06.855297 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:39:06.855274 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8008d65_ffce_4dce_8f35_011265521418.slice/crio-7db9f07446b79de5b5afbf5b69820e11ff815babd6bd4ad268a99bb2ecbc0895 WatchSource:0}: Error finding container 7db9f07446b79de5b5afbf5b69820e11ff815babd6bd4ad268a99bb2ecbc0895: Status 404 returned error can't find the container with id 7db9f07446b79de5b5afbf5b69820e11ff815babd6bd4ad268a99bb2ecbc0895 Jul 31 19:39:07.589098 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.589062 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" event={"ID":"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558","Type":"ContainerStarted","Data":"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c"} Jul 31 19:39:07.589612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.589130 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:07.589612 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.589151 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" event={"ID":"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558","Type":"ContainerStarted","Data":"18460d80035f338735aa057e21416626d7c673a514f1a1207cd0e738fb3b7715"} Jul 31 19:39:07.590689 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.590669 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" event={"ID":"d8008d65-ffce-4dce-8f35-011265521418","Type":"ContainerStarted","Data":"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532"} Jul 31 19:39:07.590769 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.590709 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" event={"ID":"d8008d65-ffce-4dce-8f35-011265521418","Type":"ContainerStarted","Data":"7db9f07446b79de5b5afbf5b69820e11ff815babd6bd4ad268a99bb2ecbc0895"} Jul 31 19:39:07.590769 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.590738 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:07.607108 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.607049 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" podStartSLOduration=1.607034978 podStartE2EDuration="1.607034978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:39:07.605422256 +0000 UTC m=+3296.143649907" watchObservedRunningTime="2026-07-31 19:39:07.607034978 +0000 UTC m=+3296.145262627" Jul 31 19:39:07.623158 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:07.623109 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" podStartSLOduration=1.623091612 podStartE2EDuration="1.623091612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:39:07.622993476 +0000 UTC m=+3296.161221126" watchObservedRunningTime="2026-07-31 19:39:07.623091612 +0000 UTC m=+3296.161319266" Jul 31 19:39:18.599397 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:18.599363 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:39:18.599840 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:18.599430 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:39:36.198167 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:36.198087 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:39:36.312135 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:36.312105 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n_d8008d65-ffce-4dce-8f35-011265521418/payload-processing/0.log" Jul 31 19:39:41.391743 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.391702 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:39:41.395661 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.395637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.399851 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.399829 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0368ff\"/\"kube-root-ca.crt\"" Jul 31 19:39:41.399971 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.399829 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0368ff\"/\"default-dockercfg-dgw6c\"" Jul 31 19:39:41.399971 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.399871 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0368ff\"/\"openshift-service-ca.crt\"" Jul 31 19:39:41.399971 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.399871 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0368ff\"/\"ipp-route-0368ff-kserve-self-signed-certs\"" Jul 31 19:39:41.404822 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.404800 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:39:41.502943 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.502912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.503155 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.502952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.503155 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.502988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.503155 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.503015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjpj\" (UniqueName: \"kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.503155 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.503043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.503155 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.503080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.604605 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.604605 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.604865 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.604865 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjpj\" (UniqueName: \"kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.604865 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.605015 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.604921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.605089 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.605066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.605145 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.605089 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.605249 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.605229 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.606977 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.606951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.607554 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.607534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.614421 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.614396 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjpj\" (UniqueName: \"kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj\") pod \"ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.706082 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.705980 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:41.849100 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:41.849076 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:39:41.850864 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:39:41.850833 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc05bf46c_9795_4613_977a_cab267d0e84a.slice/crio-25518051043de2e5754573e2fdb49c90477bd36f5923f9bb01c4c562ff4e0f9c WatchSource:0}: Error finding container 25518051043de2e5754573e2fdb49c90477bd36f5923f9bb01c4c562ff4e0f9c: Status 404 returned error can't find the container with id 25518051043de2e5754573e2fdb49c90477bd36f5923f9bb01c4c562ff4e0f9c Jul 31 19:39:42.785101 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:42.785058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" event={"ID":"c05bf46c-9795-4613-977a-cab267d0e84a","Type":"ContainerStarted","Data":"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e"} Jul 31 19:39:42.785101 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:42.785114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" event={"ID":"c05bf46c-9795-4613-977a-cab267d0e84a","Type":"ContainerStarted","Data":"25518051043de2e5754573e2fdb49c90477bd36f5923f9bb01c4c562ff4e0f9c"} Jul 31 19:39:42.805786 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:42.805727 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" podStartSLOduration=1.8057077929999998 podStartE2EDuration="1.805707793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:39:42.803251409 +0000 UTC m=+3331.341479062" watchObservedRunningTime="2026-07-31 19:39:42.805707793 +0000 UTC m=+3331.343935446" Jul 31 19:39:51.706413 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:51.706379 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:51.707054 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:51.706437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:51.719102 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:51.719073 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:39:51.847816 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:39:51.847786 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:40:15.839446 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:15.839414 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb_235d1154-d0c8-40f0-b2ee-f0fe9d70f907/payload-processing/0.log" Jul 31 19:40:15.957061 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:15.957025 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:40:16.068034 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:16.068006 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n_d8008d65-ffce-4dce-8f35-011265521418/payload-processing/0.log" Jul 31 19:40:21.717745 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.717714 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:21.721764 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.721745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.725374 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.725354 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-85e026-dockercfg-clkcm\"" Jul 31 19:40:21.725727 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.725709 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-85e026\"" Jul 31 19:40:21.734176 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.734152 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:21.802519 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.802453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.802707 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.802537 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xh7r\" (UniqueName: \"kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.903928 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.903893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.904120 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.903934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xh7r\" (UniqueName: \"kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.904970 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.904895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:21.913263 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:21.913237 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xh7r\" (UniqueName: \"kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r\") pod \"payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:22.032110 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:22.032027 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:22.178145 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:22.178110 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:22.178856 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:40:22.178823 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6c8636_c45c_40da_a1bc_510f0c26db38.slice/crio-dce6d5d919f2629a6179d1b80217be307a7f2411129ab994219790befa538cc3 WatchSource:0}: Error finding container dce6d5d919f2629a6179d1b80217be307a7f2411129ab994219790befa538cc3: Status 404 returned error can't find the container with id dce6d5d919f2629a6179d1b80217be307a7f2411129ab994219790befa538cc3 Jul 31 19:40:23.010108 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:23.010065 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" event={"ID":"ba6c8636-c45c-40da-a1bc-510f0c26db38","Type":"ContainerStarted","Data":"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f"} Jul 31 19:40:23.010650 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:23.010172 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" event={"ID":"ba6c8636-c45c-40da-a1bc-510f0c26db38","Type":"ContainerStarted","Data":"dce6d5d919f2629a6179d1b80217be307a7f2411129ab994219790befa538cc3"} Jul 31 19:40:23.010650 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:23.010203 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:23.028953 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:23.028901 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" podStartSLOduration=2.028886051 podStartE2EDuration="2.028886051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:40:23.026376705 +0000 UTC m=+3371.564604358" watchObservedRunningTime="2026-07-31 19:40:23.028886051 +0000 UTC m=+3371.567113702" Jul 31 19:40:34.018354 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:34.018308 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:51.846185 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:51.846149 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:51.846827 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:51.846389 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" podUID="ba6c8636-c45c-40da-a1bc-510f0c26db38" containerName="payload-pre-processing" containerID="cri-o://a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f" gracePeriod=30 Jul 31 19:40:52.108649 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.108587 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:52.169961 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.169932 2566 generic.go:356] "Generic (PLEG): container finished" podID="ba6c8636-c45c-40da-a1bc-510f0c26db38" containerID="a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f" exitCode=0 Jul 31 19:40:52.170138 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.169977 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" Jul 31 19:40:52.170138 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.170081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" event={"ID":"ba6c8636-c45c-40da-a1bc-510f0c26db38","Type":"ContainerDied","Data":"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f"} Jul 31 19:40:52.170138 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.170123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs" event={"ID":"ba6c8636-c45c-40da-a1bc-510f0c26db38","Type":"ContainerDied","Data":"dce6d5d919f2629a6179d1b80217be307a7f2411129ab994219790befa538cc3"} Jul 31 19:40:52.170259 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.170150 2566 scope.go:117] "RemoveContainer" containerID="a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f" Jul 31 19:40:52.179206 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.179183 2566 scope.go:117] "RemoveContainer" containerID="a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f" Jul 31 19:40:52.179479 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:40:52.179459 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f\": container with ID starting with a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f not found: ID does not exist" containerID="a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f" Jul 31 19:40:52.179574 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.179530 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f"} err="failed to get container status \"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f\": rpc error: code = NotFound desc = could not find container \"a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f\": container with ID starting with a20acebb2c19326f21310e63da2109dabe8f15bc409bedf67d4a92aa1efaaa7f not found: ID does not exist" Jul 31 19:40:52.197986 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.197956 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume\") pod \"ba6c8636-c45c-40da-a1bc-510f0c26db38\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " Jul 31 19:40:52.198090 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.198078 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xh7r\" (UniqueName: \"kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r\") pod \"ba6c8636-c45c-40da-a1bc-510f0c26db38\" (UID: \"ba6c8636-c45c-40da-a1bc-510f0c26db38\") " Jul 31 19:40:52.198333 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.198311 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba6c8636-c45c-40da-a1bc-510f0c26db38" (UID: "ba6c8636-c45c-40da-a1bc-510f0c26db38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:40:52.198408 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.198379 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6c8636-c45c-40da-a1bc-510f0c26db38-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:40:52.200370 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.200344 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r" (OuterVolumeSpecName: "kube-api-access-7xh7r") pod "ba6c8636-c45c-40da-a1bc-510f0c26db38" (UID: "ba6c8636-c45c-40da-a1bc-510f0c26db38"). InnerVolumeSpecName "kube-api-access-7xh7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:40:52.299284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.299244 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xh7r\" (UniqueName: \"kubernetes.io/projected/ba6c8636-c45c-40da-a1bc-510f0c26db38-kube-api-access-7xh7r\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:40:52.494152 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.494117 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:52.498610 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:52.498579 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-85e026-68d9bffbc6-kllqs"] Jul 31 19:40:54.036684 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:40:54.036650 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6c8636-c45c-40da-a1bc-510f0c26db38" path="/var/lib/kubelet/pods/ba6c8636-c45c-40da-a1bc-510f0c26db38/volumes" Jul 31 19:41:11.851098 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:11.851012 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:41:11.851692 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:11.851216 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" podUID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" containerName="payload-processing" containerID="cri-o://2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71" gracePeriod=30 Jul 31 19:41:11.859651 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:11.859622 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:41:11.859859 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:11.859833 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" podUID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" containerName="payload-pre-processing" containerID="cri-o://c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb" gracePeriod=30 Jul 31 19:41:12.143395 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.143372 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:41:12.146299 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.146281 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:41:12.192854 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.192819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume\") pod \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " Jul 31 19:41:12.192854 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.192860 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd6nn\" (UniqueName: \"kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn\") pod \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " Jul 31 19:41:12.193120 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.192903 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxvvx\" (UniqueName: \"kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx\") pod \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\" (UID: \"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770\") " Jul 31 19:41:12.193120 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.192942 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume\") pod \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\" (UID: \"235d1154-d0c8-40f0-b2ee-f0fe9d70f907\") " Jul 31 19:41:12.193238 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.193221 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" (UID: "e4e90b15-c3e0-4ddb-bee2-f56feb5a0770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:41:12.193618 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.193593 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "235d1154-d0c8-40f0-b2ee-f0fe9d70f907" (UID: "235d1154-d0c8-40f0-b2ee-f0fe9d70f907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:41:12.195359 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.195318 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn" (OuterVolumeSpecName: "kube-api-access-fd6nn") pod "235d1154-d0c8-40f0-b2ee-f0fe9d70f907" (UID: "235d1154-d0c8-40f0-b2ee-f0fe9d70f907"). InnerVolumeSpecName "kube-api-access-fd6nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:41:12.195591 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.195571 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx" (OuterVolumeSpecName: "kube-api-access-hxvvx") pod "e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" (UID: "e4e90b15-c3e0-4ddb-bee2-f56feb5a0770"). InnerVolumeSpecName "kube-api-access-hxvvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:41:12.283796 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.283759 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" containerID="c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb" exitCode=0 Jul 31 19:41:12.283987 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.283808 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" Jul 31 19:41:12.283987 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.283844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" event={"ID":"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770","Type":"ContainerDied","Data":"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb"} Jul 31 19:41:12.283987 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.283900 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs" event={"ID":"e4e90b15-c3e0-4ddb-bee2-f56feb5a0770","Type":"ContainerDied","Data":"9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124"} Jul 31 19:41:12.283987 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.283932 2566 scope.go:117] "RemoveContainer" containerID="c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb" Jul 31 19:41:12.285300 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.285279 2566 generic.go:356] "Generic (PLEG): container finished" podID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" containerID="2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71" exitCode=0 Jul 31 19:41:12.285403 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.285327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" event={"ID":"235d1154-d0c8-40f0-b2ee-f0fe9d70f907","Type":"ContainerDied","Data":"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71"} Jul 31 19:41:12.285403 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.285363 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" event={"ID":"235d1154-d0c8-40f0-b2ee-f0fe9d70f907","Type":"ContainerDied","Data":"550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad"} Jul 31 19:41:12.285403 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.285328 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb" Jul 31 19:41:12.294427 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294221 2566 scope.go:117] "RemoveContainer" containerID="c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb" Jul 31 19:41:12.294427 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294401 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:12.294638 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294427 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:12.294638 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294443 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6nn\" (UniqueName: \"kubernetes.io/projected/235d1154-d0c8-40f0-b2ee-f0fe9d70f907-kube-api-access-fd6nn\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:12.294638 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294460 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvvx\" (UniqueName: \"kubernetes.io/projected/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770-kube-api-access-hxvvx\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:12.294805 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:41:12.294673 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb\": container with ID starting with c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb not found: ID does not exist" containerID="c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb" Jul 31 19:41:12.294805 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294706 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb"} err="failed to get container status \"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb\": rpc error: code = NotFound desc = could not find container \"c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb\": container with ID starting with c122c0bc32b9601f5e4e600804efd1cc8eb6395381218421ee3ec7ae787078bb not found: ID does not exist" Jul 31 19:41:12.294805 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.294727 2566 scope.go:117] "RemoveContainer" containerID="2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71" Jul 31 19:41:12.303724 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.303709 2566 scope.go:117] "RemoveContainer" containerID="2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71" Jul 31 19:41:12.304019 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:41:12.303998 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71\": container with ID starting with 2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71 not found: ID does not exist" containerID="2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71" Jul 31 19:41:12.304123 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.304024 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71"} err="failed to get container status \"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71\": rpc error: code = NotFound desc = could not find container \"2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71\": container with ID starting with 2e385f802c6ef0c848e9773ea90aa9170cebdb1666c9813da7509bb60f7c9f71 not found: ID does not exist" Jul 31 19:41:12.311109 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.311075 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:41:12.315456 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.315431 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0368ff-6b4d4b5c96-tmwqs"] Jul 31 19:41:12.328135 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.328110 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:41:12.332475 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:12.332455 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0368ff-695bc8587b-kjqxb"] Jul 31 19:41:12.335043 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:41:12.335001 2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e90b15_c3e0_4ddb_bee2_f56feb5a0770.slice/crio-9d23daabdcb1fea260811061846f279b40cb6c9ee40a738c8ba375d771f86124\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e90b15_c3e0_4ddb_bee2_f56feb5a0770.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d1154_d0c8_40f0_b2ee_f0fe9d70f907.slice/crio-550c88d77273ce88e7f43d5e870869b6446c0ad710ad88a8f9f0cdc6f0d566ad\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d1154_d0c8_40f0_b2ee_f0fe9d70f907.slice\": RecentStats: unable to find data in memory cache]" Jul 31 19:41:14.035248 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:14.035216 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" path="/var/lib/kubelet/pods/235d1154-d0c8-40f0-b2ee-f0fe9d70f907/volumes" Jul 31 19:41:14.035653 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:14.035579 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" path="/var/lib/kubelet/pods/e4e90b15-c3e0-4ddb-bee2-f56feb5a0770/volumes" Jul 31 19:41:17.348333 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:17.348294 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:41:32.368777 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.368728 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" podUID="c05bf46c-9795-4613-977a-cab267d0e84a" containerName="main" containerID="cri-o://c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e" gracePeriod=45 Jul 31 19:41:32.620307 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.620170 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:41:32.690764 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.690730 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.690961 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.690837 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.690961 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.690876 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.690961 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.690917 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.690961 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.690953 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.691188 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691012 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:32.691188 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691037 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:32.691188 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691045 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjpj\" (UniqueName: \"kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj\") pod \"c05bf46c-9795-4613-977a-cab267d0e84a\" (UID: \"c05bf46c-9795-4613-977a-cab267d0e84a\") " Jul 31 19:41:32.691364 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691204 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home" (OuterVolumeSpecName: "home") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:32.691418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691388 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-model-cache\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:32.691418 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691404 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-tmp-dir\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:32.691554 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.691418 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-home\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:32.693794 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.693767 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 19:41:32.693914 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.693822 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj" (OuterVolumeSpecName: "kube-api-access-fkjpj") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "kube-api-access-fkjpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:41:32.693914 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.693835 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm" (OuterVolumeSpecName: "dshm") pod "c05bf46c-9795-4613-977a-cab267d0e84a" (UID: "c05bf46c-9795-4613-977a-cab267d0e84a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 19:41:32.792756 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.792712 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05bf46c-9795-4613-977a-cab267d0e84a-dshm\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:32.792756 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.792748 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05bf46c-9795-4613-977a-cab267d0e84a-tls-certs\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:32.792756 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:32.792765 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjpj\" (UniqueName: \"kubernetes.io/projected/c05bf46c-9795-4613-977a-cab267d0e84a-kube-api-access-fkjpj\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:41:33.396705 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.396670 2566 generic.go:356] "Generic (PLEG): container finished" podID="c05bf46c-9795-4613-977a-cab267d0e84a" containerID="c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e" exitCode=0 Jul 31 19:41:33.397161 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.396713 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" Jul 31 19:41:33.397161 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.396750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" event={"ID":"c05bf46c-9795-4613-977a-cab267d0e84a","Type":"ContainerDied","Data":"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e"} Jul 31 19:41:33.397161 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.396791 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s" event={"ID":"c05bf46c-9795-4613-977a-cab267d0e84a","Type":"ContainerDied","Data":"25518051043de2e5754573e2fdb49c90477bd36f5923f9bb01c4c562ff4e0f9c"} Jul 31 19:41:33.397161 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.396808 2566 scope.go:117] "RemoveContainer" containerID="c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e" Jul 31 19:41:33.406030 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.406009 2566 scope.go:117] "RemoveContainer" containerID="c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e" Jul 31 19:41:33.406278 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:41:33.406258 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e\": container with ID starting with c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e not found: ID does not exist" containerID="c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e" Jul 31 19:41:33.406352 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.406285 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e"} err="failed to get container status \"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e\": rpc error: code = NotFound desc = could not find container \"c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e\": container with ID starting with c70f83945f44cf3cba01b72226770619dadf8a53bf02973c1aea6307093bcb4e not found: ID does not exist" Jul 31 19:41:33.420519 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.420467 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:41:33.424905 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:33.424884 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0368ff/ipp-route-0368ff-kserve-6dcd97d8dc-zhw2s"] Jul 31 19:41:34.035679 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:41:34.035646 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05bf46c-9795-4613-977a-cab267d0e84a" path="/var/lib/kubelet/pods/c05bf46c-9795-4613-977a-cab267d0e84a/volumes" Jul 31 19:42:17.587283 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.587235 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:42:17.587769 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.587533 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" podUID="d8008d65-ffce-4dce-8f35-011265521418" containerName="payload-processing" containerID="cri-o://78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532" gracePeriod=30 Jul 31 19:42:17.594730 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.592389 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:42:17.594730 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.593595 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" podUID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" containerName="payload-pre-processing" containerID="cri-o://c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c" gracePeriod=30 Jul 31 19:42:17.849347 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.849288 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:42:17.879670 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.879648 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:42:17.940856 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.940827 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume\") pod \"d8008d65-ffce-4dce-8f35-011265521418\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " Jul 31 19:42:17.941060 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.940962 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsmwc\" (UniqueName: \"kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc\") pod \"d8008d65-ffce-4dce-8f35-011265521418\" (UID: \"d8008d65-ffce-4dce-8f35-011265521418\") " Jul 31 19:42:17.941244 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.941220 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8008d65-ffce-4dce-8f35-011265521418" (UID: "d8008d65-ffce-4dce-8f35-011265521418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:17.941381 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.941366 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8008d65-ffce-4dce-8f35-011265521418-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:17.943174 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:17.943153 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc" (OuterVolumeSpecName: "kube-api-access-jsmwc") pod "d8008d65-ffce-4dce-8f35-011265521418" (UID: "d8008d65-ffce-4dce-8f35-011265521418"). InnerVolumeSpecName "kube-api-access-jsmwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:18.041710 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.041681 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqnjj\" (UniqueName: \"kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj\") pod \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " Jul 31 19:42:18.041846 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.041728 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume\") pod \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\" (UID: \"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558\") " Jul 31 19:42:18.041965 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.041953 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmwc\" (UniqueName: \"kubernetes.io/projected/d8008d65-ffce-4dce-8f35-011265521418-kube-api-access-jsmwc\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:18.042147 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.042125 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" (UID: "5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:18.043970 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.043947 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj" (OuterVolumeSpecName: "kube-api-access-tqnjj") pod "5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" (UID: "5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558"). InnerVolumeSpecName "kube-api-access-tqnjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:18.142530 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.142467 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnjj\" (UniqueName: \"kubernetes.io/projected/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-kube-api-access-tqnjj\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:18.142710 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.142535 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:18.636053 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.636018 2566 generic.go:356] "Generic (PLEG): container finished" podID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" containerID="c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c" exitCode=0 Jul 31 19:42:18.636549 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.636057 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" Jul 31 19:42:18.636549 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.636080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" event={"ID":"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558","Type":"ContainerDied","Data":"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c"} Jul 31 19:42:18.636549 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.636136 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl" event={"ID":"5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558","Type":"ContainerDied","Data":"18460d80035f338735aa057e21416626d7c673a514f1a1207cd0e738fb3b7715"} Jul 31 19:42:18.636549 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.636164 2566 scope.go:117] "RemoveContainer" containerID="c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c" Jul 31 19:42:18.637563 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.637545 2566 generic.go:356] "Generic (PLEG): container finished" podID="d8008d65-ffce-4dce-8f35-011265521418" containerID="78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532" exitCode=0 Jul 31 19:42:18.637631 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.637603 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" Jul 31 19:42:18.637631 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.637613 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" event={"ID":"d8008d65-ffce-4dce-8f35-011265521418","Type":"ContainerDied","Data":"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532"} Jul 31 19:42:18.637712 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.637652 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n" event={"ID":"d8008d65-ffce-4dce-8f35-011265521418","Type":"ContainerDied","Data":"7db9f07446b79de5b5afbf5b69820e11ff815babd6bd4ad268a99bb2ecbc0895"} Jul 31 19:42:18.645623 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.645608 2566 scope.go:117] "RemoveContainer" containerID="c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c" Jul 31 19:42:18.645870 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:42:18.645851 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c\": container with ID starting with c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c not found: ID does not exist" containerID="c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c" Jul 31 19:42:18.645914 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.645879 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c"} err="failed to get container status \"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c\": rpc error: code = NotFound desc = could not find container \"c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c\": container with ID starting with c03520aaf2159552655e41d00187415a6eee5a91963a385d3c5281eba17db92c not found: ID does not exist" Jul 31 19:42:18.645914 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.645894 2566 scope.go:117] "RemoveContainer" containerID="78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532" Jul 31 19:42:18.653789 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.653772 2566 scope.go:117] "RemoveContainer" containerID="78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532" Jul 31 19:42:18.654025 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:42:18.654004 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532\": container with ID starting with 78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532 not found: ID does not exist" containerID="78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532" Jul 31 19:42:18.654093 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.654035 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532"} err="failed to get container status \"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532\": rpc error: code = NotFound desc = could not find container \"78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532\": container with ID starting with 78bf2764fed8e24da4bbee1d7c2933d4a2c9a2b3b1f8ea44f6c8ef70856c5532 not found: ID does not exist" Jul 31 19:42:18.658400 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.658377 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:42:18.662360 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.662339 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90f898-cc8548f44-wgb2n"] Jul 31 19:42:18.674112 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.674090 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:42:18.678382 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:18.678361 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90f898-54d994c8d8-xxjpl"] Jul 31 19:42:20.035335 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:20.035300 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" path="/var/lib/kubelet/pods/5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558/volumes" Jul 31 19:42:20.035732 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:20.035667 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8008d65-ffce-4dce-8f35-011265521418" path="/var/lib/kubelet/pods/d8008d65-ffce-4dce-8f35-011265521418/volumes" Jul 31 19:42:24.821236 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821205 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9"] Jul 31 19:42:24.821680 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821666 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" containerName="payload-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821682 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" containerName="payload-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821691 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821697 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821711 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821718 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821725 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6c8636-c45c-40da-a1bc-510f0c26db38" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821729 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6c8636-c45c-40da-a1bc-510f0c26db38" containerName="payload-pre-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821737 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8008d65-ffce-4dce-8f35-011265521418" containerName="payload-processing" Jul 31 19:42:24.821741 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821744 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8008d65-ffce-4dce-8f35-011265521418" containerName="payload-processing" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821754 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05bf46c-9795-4613-977a-cab267d0e84a" containerName="main" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821759 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05bf46c-9795-4613-977a-cab267d0e84a" containerName="main" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821828 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8008d65-ffce-4dce-8f35-011265521418" containerName="payload-processing" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821835 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05bf46c-9795-4613-977a-cab267d0e84a" containerName="main" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821841 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e90b15-c3e0-4ddb-bee2-f56feb5a0770" containerName="payload-pre-processing" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821846 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6c8636-c45c-40da-a1bc-510f0c26db38" containerName="payload-pre-processing" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821853 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb7c412-bb2b-4d6a-9cf7-edf0b31ab558" containerName="payload-pre-processing" Jul 31 19:42:24.822013 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.821859 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="235d1154-d0c8-40f0-b2ee-f0fe9d70f907" containerName="payload-processing" Jul 31 19:42:24.824734 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.824715 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:24.827357 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.827329 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\"" Jul 31 19:42:24.834428 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.834403 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9"] Jul 31 19:42:24.902781 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:24.902744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9\" (UID: \"be527b4e-f9c1-4b03-8118-9880fcccba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:25.003640 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.003601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9\" (UID: \"be527b4e-f9c1-4b03-8118-9880fcccba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:25.004201 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.004179 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9\" (UID: \"be527b4e-f9c1-4b03-8118-9880fcccba37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:25.136981 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.136897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:25.281682 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.281659 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9"] Jul 31 19:42:25.283822 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:42:25.283796 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe527b4e_f9c1_4b03_8118_9880fcccba37.slice/crio-56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd WatchSource:0}: Error finding container 56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd: Status 404 returned error can't find the container with id 56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd Jul 31 19:42:25.675805 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.675770 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" event={"ID":"be527b4e-f9c1-4b03-8118-9880fcccba37","Type":"ContainerStarted","Data":"4723a3d6384954f86c1fe8b72f4e6c677f69e26504868944335fdfa6297d5a2b"} Jul 31 19:42:25.675986 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.675820 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" event={"ID":"be527b4e-f9c1-4b03-8118-9880fcccba37","Type":"ContainerStarted","Data":"56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd"} Jul 31 19:42:25.697526 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:25.697461 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" podStartSLOduration=1.69744385 podStartE2EDuration="1.69744385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:42:25.69552554 +0000 UTC m=+3494.233753186" watchObservedRunningTime="2026-07-31 19:42:25.69744385 +0000 UTC m=+3494.235671673" Jul 31 19:42:26.682189 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:26.682160 2566 generic.go:356] "Generic (PLEG): container finished" podID="be527b4e-f9c1-4b03-8118-9880fcccba37" containerID="4723a3d6384954f86c1fe8b72f4e6c677f69e26504868944335fdfa6297d5a2b" exitCode=0 Jul 31 19:42:26.682591 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:26.682227 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" event={"ID":"be527b4e-f9c1-4b03-8118-9880fcccba37","Type":"ContainerDied","Data":"4723a3d6384954f86c1fe8b72f4e6c677f69e26504868944335fdfa6297d5a2b"} Jul 31 19:42:27.822406 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:27.822381 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:27.826447 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:27.826429 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca\") pod \"be527b4e-f9c1-4b03-8118-9880fcccba37\" (UID: \"be527b4e-f9c1-4b03-8118-9880fcccba37\") " Jul 31 19:42:27.826780 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:27.826744 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be527b4e-f9c1-4b03-8118-9880fcccba37" (UID: "be527b4e-f9c1-4b03-8118-9880fcccba37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:27.927179 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:27.927146 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be527b4e-f9c1-4b03-8118-9880fcccba37-maas-api-service-ca\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:28.694475 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:28.694443 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" Jul 31 19:42:28.694673 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:28.694484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9" event={"ID":"be527b4e-f9c1-4b03-8118-9880fcccba37","Type":"ContainerDied","Data":"56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd"} Jul 31 19:42:28.694673 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:28.694553 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56b33e91a66fa655d8abbb8beda118d90b8896cf5d3a6af99ea988aa86c126dd" Jul 31 19:42:35.612650 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:35.612567 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9"] Jul 31 19:42:35.615511 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:35.615465 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd0284-pf2l9"] Jul 31 19:42:36.035648 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:36.035613 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be527b4e-f9c1-4b03-8118-9880fcccba37" path="/var/lib/kubelet/pods/be527b4e-f9c1-4b03-8118-9880fcccba37/volumes" Jul 31 19:42:53.518133 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.518096 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:42:53.518750 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.518345 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" podUID="655b4fdb-7939-4fbc-9034-9c5995ede111" containerName="payload-pre-processing" containerID="cri-o://e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f" gracePeriod=30 Jul 31 19:42:53.778270 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.778214 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:42:53.836234 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.836193 2566 generic.go:356] "Generic (PLEG): container finished" podID="655b4fdb-7939-4fbc-9034-9c5995ede111" containerID="e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f" exitCode=0 Jul 31 19:42:53.836395 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.836253 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" Jul 31 19:42:53.836395 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.836289 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" event={"ID":"655b4fdb-7939-4fbc-9034-9c5995ede111","Type":"ContainerDied","Data":"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f"} Jul 31 19:42:53.836395 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.836368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm" event={"ID":"655b4fdb-7939-4fbc-9034-9c5995ede111","Type":"ContainerDied","Data":"2cb943ec233cf31814f90996dffce56acea5d76f58380899f733d140a7647da9"} Jul 31 19:42:53.836395 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.836392 2566 scope.go:117] "RemoveContainer" containerID="e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f" Jul 31 19:42:53.845157 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.845139 2566 scope.go:117] "RemoveContainer" containerID="e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f" Jul 31 19:42:53.845423 ip-10-0-139-43 kubenswrapper[2566]: E0731 19:42:53.845403 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f\": container with ID starting with e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f not found: ID does not exist" containerID="e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f" Jul 31 19:42:53.845467 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.845432 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f"} err="failed to get container status \"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f\": rpc error: code = NotFound desc = could not find container \"e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f\": container with ID starting with e7978bcb716e2cce5c6e0a2b33ccceddb2c029bf75b88eca414879d8cf37078f not found: ID does not exist" Jul 31 19:42:53.884289 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.884260 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume\") pod \"655b4fdb-7939-4fbc-9034-9c5995ede111\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " Jul 31 19:42:53.884421 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.884410 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgbg5\" (UniqueName: \"kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5\") pod \"655b4fdb-7939-4fbc-9034-9c5995ede111\" (UID: \"655b4fdb-7939-4fbc-9034-9c5995ede111\") " Jul 31 19:42:53.884716 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.884687 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "655b4fdb-7939-4fbc-9034-9c5995ede111" (UID: "655b4fdb-7939-4fbc-9034-9c5995ede111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 19:42:53.886639 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.886613 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5" (OuterVolumeSpecName: "kube-api-access-kgbg5") pod "655b4fdb-7939-4fbc-9034-9c5995ede111" (UID: "655b4fdb-7939-4fbc-9034-9c5995ede111"). InnerVolumeSpecName "kube-api-access-kgbg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 19:42:53.986075 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.986037 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b4fdb-7939-4fbc-9034-9c5995ede111-plugins-config-volume\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:53.986075 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:53.986068 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbg5\" (UniqueName: \"kubernetes.io/projected/655b4fdb-7939-4fbc-9034-9c5995ede111-kube-api-access-kgbg5\") on node \"ip-10-0-139-43.ec2.internal\" DevicePath \"\"" Jul 31 19:42:54.156256 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:54.156219 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:42:54.160371 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:54.160350 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-09a154-f8c5f94c6-dzgxm"] Jul 31 19:42:56.035302 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:42:56.035262 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655b4fdb-7939-4fbc-9034-9c5995ede111" path="/var/lib/kubelet/pods/655b4fdb-7939-4fbc-9034-9c5995ede111/volumes" Jul 31 19:43:06.433249 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:06.433215 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5667fb5f9d-clqzj_93253fe4-60ab-4ac5-9285-f122558b6da2/manager/0.log" Jul 31 19:43:07.020078 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.020047 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-zk8c9_288ea94a-9419-4a30-94ce-bafd8ccd8c63/copy-manifests/0.log" Jul 31 19:43:07.027682 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.027655 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-zk8c9_288ea94a-9419-4a30-94ce-bafd8ccd8c63/manager/0.log" Jul 31 19:43:07.369028 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.368953 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c89d6c77-dqb6q_7bb9cd45-7a31-4785-b24f-9353ca06fa3c/maas-api/0.log" Jul 31 19:43:07.474737 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.474710 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758740-tkpg7_2d77faa5-0282-46d3-9c3d-acf78421ba2b/cleanup/0.log" Jul 31 19:43:07.684755 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.684727 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758770-7545f_92ee67c9-e549-4a00-862b-18a7302db232/cleanup/0.log" Jul 31 19:43:07.791091 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:07.791053 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xtszm_8a4b6a1a-a2a1-48cd-bab3-eb7d72d6d935/postgres/0.log" Jul 31 19:43:08.973738 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:08.973711 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-fnnxv_95a1dbc3-9d82-43df-98b5-5abfba9d5fa7/manager/0.log" Jul 31 19:43:09.858525 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:09.858475 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-phcjk_85dcf311-dd28-4e35-ab25-302504bc80b7/kube-auth-proxy/0.log" Jul 31 19:43:10.088547 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:10.088517 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fmczj_997605d3-a1e0-4855-a57b-5294c87a4efb/payload-pre-processing/0.log" Jul 31 19:43:10.206544 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:10.206514 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:43:10.747625 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:10.747589 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-mmb92_dc7d6ec6-5e3f-4cd1-bdcb-191e36f59a4f/main/0.log" Jul 31 19:43:10.855332 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:10.855302 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rf6b2_ffcd3813-4be7-4e3b-ba23-c639ce1eb702/main/0.log" Jul 31 19:43:19.520459 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:19.520424 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6gr7q_b178f3a3-fa58-4d04-9e33-bf0533dc0aea/global-pull-secret-syncer/0.log" Jul 31 19:43:19.701717 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:19.701681 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-whppk_fe239cdd-811b-4875-92d6-21ecf3c41138/konnectivity-agent/0.log" Jul 31 19:43:19.783412 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:19.783302 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-43.ec2.internal_63165082ff7f5713045b9ebffd02955c/haproxy/0.log" Jul 31 19:43:25.168158 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:25.168120 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-fnnxv_95a1dbc3-9d82-43df-98b5-5abfba9d5fa7/manager/0.log" Jul 31 19:43:26.764879 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.764833 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/alertmanager/0.log" Jul 31 19:43:26.794866 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.794828 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/config-reloader/0.log" Jul 31 19:43:26.831381 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.831300 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/kube-rbac-proxy-web/0.log" Jul 31 19:43:26.865439 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.865405 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/kube-rbac-proxy/0.log" Jul 31 19:43:26.890453 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.890415 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/kube-rbac-proxy-metric/0.log" Jul 31 19:43:26.916069 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.916028 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/prom-label-proxy/0.log" Jul 31 19:43:26.942241 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:26.942210 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee905a3d-0486-4d25-9af4-f79bf5c6d8b5/init-config-reloader/0.log" Jul 31 19:43:27.031810 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.031780 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-skr92_2663c406-3579-4757-96dc-3a79b46fdc2b/kube-state-metrics/0.log" Jul 31 19:43:27.051781 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.051746 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-skr92_2663c406-3579-4757-96dc-3a79b46fdc2b/kube-rbac-proxy-main/0.log" Jul 31 19:43:27.076811 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.076785 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-skr92_2663c406-3579-4757-96dc-3a79b46fdc2b/kube-rbac-proxy-self/0.log" Jul 31 19:43:27.108243 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.108152 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bdcd74c68-vfbcj_ac265491-b08e-4d17-8512-3060b343591b/metrics-server/0.log" Jul 31 19:43:27.141400 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.141370 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-t8jng_5140d648-06e5-4177-a382-9fac12613660/monitoring-plugin/0.log" Jul 31 19:43:27.353592 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.353565 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccppf_7245fb16-fdce-4d85-9f58-6c2dfc5dade6/node-exporter/0.log" Jul 31 19:43:27.373931 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.373861 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccppf_7245fb16-fdce-4d85-9f58-6c2dfc5dade6/kube-rbac-proxy/0.log" Jul 31 19:43:27.396909 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.396883 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccppf_7245fb16-fdce-4d85-9f58-6c2dfc5dade6/init-textfile/0.log" Jul 31 19:43:27.430744 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.430714 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-8ltzf_0a524a1b-6de2-425f-84fc-5d7048f4983a/kube-rbac-proxy-main/0.log" Jul 31 19:43:27.459060 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.459032 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-8ltzf_0a524a1b-6de2-425f-84fc-5d7048f4983a/kube-rbac-proxy-self/0.log" Jul 31 19:43:27.482038 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.482013 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-8ltzf_0a524a1b-6de2-425f-84fc-5d7048f4983a/openshift-state-metrics/0.log" Jul 31 19:43:27.521497 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.521470 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/prometheus/0.log" Jul 31 19:43:27.548221 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.548194 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/config-reloader/0.log" Jul 31 19:43:27.571474 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.571440 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/thanos-sidecar/0.log" Jul 31 19:43:27.600387 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.600360 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/kube-rbac-proxy-web/0.log" Jul 31 19:43:27.626758 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.626680 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/kube-rbac-proxy/0.log" Jul 31 19:43:27.651751 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.651720 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/kube-rbac-proxy-thanos/0.log" Jul 31 19:43:27.677747 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.677717 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0aaffbe-ca4c-4ab4-aa8f-92bf06c15107/init-config-reloader/0.log" Jul 31 19:43:27.710985 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.710955 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-gg8zs_7f3ac65c-394a-4b33-ae59-6148d7853d46/prometheus-operator/0.log" Jul 31 19:43:27.730888 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.730859 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-gg8zs_7f3ac65c-394a-4b33-ae59-6148d7853d46/kube-rbac-proxy/0.log" Jul 31 19:43:27.756817 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.756787 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-qf2hg_f3fe4c28-933d-4370-a3d3-6271a4764176/prometheus-operator-admission-webhook/0.log" Jul 31 19:43:27.788273 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.788245 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5b9968c5-h9rhr_cccaa60f-50b0-4cf6-9fcd-8974586dfe18/telemeter-client/0.log" Jul 31 19:43:27.811776 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.811751 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5b9968c5-h9rhr_cccaa60f-50b0-4cf6-9fcd-8974586dfe18/reload/0.log" Jul 31 19:43:27.838990 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.838963 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5b9968c5-h9rhr_cccaa60f-50b0-4cf6-9fcd-8974586dfe18/kube-rbac-proxy/0.log" Jul 31 19:43:27.878151 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.878073 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/thanos-query/0.log" Jul 31 19:43:27.900780 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.900749 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/kube-rbac-proxy-web/0.log" Jul 31 19:43:27.923480 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.923450 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/kube-rbac-proxy/0.log" Jul 31 19:43:27.947969 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.947942 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/prom-label-proxy/0.log" Jul 31 19:43:27.970374 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.970342 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/kube-rbac-proxy-rules/0.log" Jul 31 19:43:27.992169 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:27.992141 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b974ff869-jj6t6_6cc6b4fd-55fa-4c1b-a360-e20775d90b63/kube-rbac-proxy-metrics/0.log" Jul 31 19:43:29.545855 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.545818 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw"] Jul 31 19:43:29.546298 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546281 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655b4fdb-7939-4fbc-9034-9c5995ede111" containerName="payload-pre-processing" Jul 31 19:43:29.546342 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546302 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="655b4fdb-7939-4fbc-9034-9c5995ede111" containerName="payload-pre-processing" Jul 31 19:43:29.546342 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546333 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be527b4e-f9c1-4b03-8118-9880fcccba37" containerName="revoke-keys" Jul 31 19:43:29.546342 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546338 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="be527b4e-f9c1-4b03-8118-9880fcccba37" containerName="revoke-keys" Jul 31 19:43:29.546439 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546405 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="655b4fdb-7939-4fbc-9034-9c5995ede111" containerName="payload-pre-processing" Jul 31 19:43:29.546439 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.546415 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="be527b4e-f9c1-4b03-8118-9880fcccba37" containerName="revoke-keys" Jul 31 19:43:29.550040 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.550019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.553258 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.553240 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"kube-root-ca.crt\"" Jul 31 19:43:29.554642 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.554608 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"openshift-service-ca.crt\"" Jul 31 19:43:29.554780 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.554762 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x2d7j\"/\"default-dockercfg-kfhf6\"" Jul 31 19:43:29.558250 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.558215 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw"] Jul 31 19:43:29.614880 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.614841 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-proc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.615024 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.614883 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-sys\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.615024 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.614953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-podres\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.615024 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.614998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddxc\" (UniqueName: \"kubernetes.io/projected/23daabc5-44c4-43cd-8e84-195fa257ae86-kube-api-access-2ddxc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.615147 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.615026 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-lib-modules\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716082 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-proc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716088 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-sys\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716131 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-podres\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716170 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddxc\" (UniqueName: \"kubernetes.io/projected/23daabc5-44c4-43cd-8e84-195fa257ae86-kube-api-access-2ddxc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716172 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-proc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-lib-modules\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716284 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716228 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-sys\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716585 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716285 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-podres\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.716585 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.716362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23daabc5-44c4-43cd-8e84-195fa257ae86-lib-modules\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.727165 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.727136 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddxc\" (UniqueName: \"kubernetes.io/projected/23daabc5-44c4-43cd-8e84-195fa257ae86-kube-api-access-2ddxc\") pod \"perf-node-gather-daemonset-bzhkw\" (UID: \"23daabc5-44c4-43cd-8e84-195fa257ae86\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:29.862127 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:29.862024 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:30.000378 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:30.000341 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw"] Jul 31 19:43:30.002244 ip-10-0-139-43 kubenswrapper[2566]: W0731 19:43:30.002212 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23daabc5_44c4_43cd_8e84_195fa257ae86.slice/crio-bd031600794a67d697d49dd1e39e5a04813b223d436671c27fc5c56bd89e7f4c WatchSource:0}: Error finding container bd031600794a67d697d49dd1e39e5a04813b223d436671c27fc5c56bd89e7f4c: Status 404 returned error can't find the container with id bd031600794a67d697d49dd1e39e5a04813b223d436671c27fc5c56bd89e7f4c Jul 31 19:43:30.022828 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:30.022800 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" event={"ID":"23daabc5-44c4-43cd-8e84-195fa257ae86","Type":"ContainerStarted","Data":"bd031600794a67d697d49dd1e39e5a04813b223d436671c27fc5c56bd89e7f4c"} Jul 31 19:43:31.030251 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:31.030211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" event={"ID":"23daabc5-44c4-43cd-8e84-195fa257ae86","Type":"ContainerStarted","Data":"6acdb7897ffc72915d29c15630c2f2d7c3cc222a580c975ba3347f9cf189662e"} Jul 31 19:43:31.030770 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:31.030391 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:31.050636 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:31.050579 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" podStartSLOduration=2.050564525 podStartE2EDuration="2.050564525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:43:31.048015546 +0000 UTC m=+3559.586243207" watchObservedRunningTime="2026-07-31 19:43:31.050564525 +0000 UTC m=+3559.588792175" Jul 31 19:43:32.360066 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:32.360033 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jd6f_c9a045f5-fe8e-4839-b71c-f9c671da2cae/dns/0.log" Jul 31 19:43:32.383162 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:32.383132 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jd6f_c9a045f5-fe8e-4839-b71c-f9c671da2cae/kube-rbac-proxy/0.log" Jul 31 19:43:32.511589 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:32.511560 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tcvq_996cb3b4-342a-44e6-8e35-60887aab2638/dns-node-resolver/0.log" Jul 31 19:43:33.216607 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:33.216579 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zt6dh_242f0489-8ae7-42e5-bb8b-77d84a5fa711/node-ca/0.log" Jul 31 19:43:34.408291 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:34.408264 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-phcjk_85dcf311-dd28-4e35-ab25-302504bc80b7/kube-auth-proxy/0.log" Jul 31 19:43:34.486939 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:34.486907 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fmczj_997605d3-a1e0-4855-a57b-5294c87a4efb/payload-pre-processing/0.log" Jul 31 19:43:34.531386 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:34.531356 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:43:35.363719 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:35.363690 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bh6g_a5014d93-4ec1-4abd-a819-cdc21e81722e/serve-healthcheck-canary/0.log" Jul 31 19:43:35.985070 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:35.985031 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5l2mc_4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970/kube-rbac-proxy/0.log" Jul 31 19:43:36.009228 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:36.009202 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5l2mc_4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970/exporter/0.log" Jul 31 19:43:36.035340 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:36.035313 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5l2mc_4f8a1a5a-6752-4c8e-ab3b-8e0a5868b970/extractor/0.log" Jul 31 19:43:37.045369 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:37.045337 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-bzhkw" Jul 31 19:43:38.954366 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:38.954333 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5667fb5f9d-clqzj_93253fe4-60ab-4ac5-9285-f122558b6da2/manager/0.log" Jul 31 19:43:39.286455 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:39.286426 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-zk8c9_288ea94a-9419-4a30-94ce-bafd8ccd8c63/manager/0.log" Jul 31 19:43:39.312470 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:39.312443 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-zk8c9_288ea94a-9419-4a30-94ce-bafd8ccd8c63/copy-manifests/0.log" Jul 31 19:43:40.220721 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:40.220694 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-phcjk_85dcf311-dd28-4e35-ab25-302504bc80b7/kube-auth-proxy/0.log" Jul 31 19:43:40.285955 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:40.285917 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fmczj_997605d3-a1e0-4855-a57b-5294c87a4efb/payload-pre-processing/0.log" Jul 31 19:43:40.318012 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:40.317977 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:43:40.869890 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:40.869860 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6478f7cf66-kht26_a209c091-7485-4aaf-a01d-cd80d74952c6/manager/0.log" Jul 31 19:43:42.008763 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:42.008727 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dbc777-phcjk_85dcf311-dd28-4e35-ab25-302504bc80b7/kube-auth-proxy/0.log" Jul 31 19:43:42.073340 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:42.073256 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fmczj_997605d3-a1e0-4855-a57b-5294c87a4efb/payload-pre-processing/0.log" Jul 31 19:43:42.103953 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:42.103919 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-fl624_f89da6e1-91b4-49a2-84f6-b89ac08918c6/payload-processing/0.log" Jul 31 19:43:47.868684 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.868657 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/kube-multus-additional-cni-plugins/0.log" Jul 31 19:43:47.890110 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.890078 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/egress-router-binary-copy/0.log" Jul 31 19:43:47.911512 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.911458 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/cni-plugins/0.log" Jul 31 19:43:47.934555 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.934526 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/bond-cni-plugin/0.log" Jul 31 19:43:47.960008 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.959977 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/routeoverride-cni/0.log" Jul 31 19:43:47.982817 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:47.982793 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/whereabouts-cni-bincopy/0.log" Jul 31 19:43:48.005085 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:48.005051 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lx6kw_a67104f9-4e22-41f3-aa18-5f401c8503be/whereabouts-cni/0.log" Jul 31 19:43:48.249907 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:48.249869 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gmpwb_5c44670c-587b-4929-8898-672eb5b2ad8b/kube-multus/0.log" Jul 31 19:43:48.367519 ip-10-0-139-43 kubenswrapper[2566]: I0731 19:43:48.367477 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nswnl_82292e4a-5143-4fd1-8750-b8502a4041d6/network-metrics-daemon/0.log"