Jul 31 22:56:51.965045 ip-10-0-137-58 systemd[1]: Starting Kubernetes Kubelet... Jul 31 22:56:52.347895 ip-10-0-137-58 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 22:56:52.347895 ip-10-0-137-58 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 22:56:52.347895 ip-10-0-137-58 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 22:56:52.347895 ip-10-0-137-58 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 22:56:52.347895 ip-10-0-137-58 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 22:56:52.348496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.347916 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 22:56:52.352085 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352070 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.352085 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352083 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352088 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352092 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352095 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352098 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352100 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352103 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352106 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352117 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352120 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352122 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352125 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352128 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352131 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352133 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352136 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352138 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352141 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352143 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352146 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.352150 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352148 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352151 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352153 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352156 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352160 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352163 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352166 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352168 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352171 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352173 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352176 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352178 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352181 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352183 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352186 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352188 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352191 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352194 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352196 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352198 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.352598 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352201 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352203 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352205 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352208 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352210 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352212 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352220 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352223 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352227 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352231 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352234 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352238 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352240 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352243 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352247 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352250 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352252 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352255 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352258 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.353094 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352260 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352262 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352265 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352268 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352271 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352273 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352275 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352278 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352281 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352283 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352286 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352288 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352290 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352293 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352298 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352301 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352304 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352307 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352309 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352312 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.353550 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352314 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352317 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352319 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352322 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352324 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352326 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352329 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352332 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352335 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352338 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352340 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352343 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352345 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352348 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352350 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352353 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352355 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352358 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.352360 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353732 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.354065 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353740 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353744 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353747 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353750 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353753 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353755 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353758 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353761 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353763 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353766 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353768 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353771 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353774 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353776 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353779 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353782 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353784 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353787 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353789 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353793 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.354524 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353796 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353799 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353801 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353804 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353808 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353812 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353814 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353829 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353832 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353835 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353838 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353840 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353843 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353845 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353848 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353850 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353853 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353856 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353858 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.355011 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353861 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353863 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353866 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353868 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353870 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353873 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353875 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353878 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353881 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353885 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353888 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353890 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353893 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353896 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353899 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353902 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353904 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353907 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353909 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.355454 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353912 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353914 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353917 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353919 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353922 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353924 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353927 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353929 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353931 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353934 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353936 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353939 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353941 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353944 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353947 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353952 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353955 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353958 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353960 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.355913 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353963 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353966 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353968 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353971 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353973 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353976 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353979 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353982 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353984 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353987 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353989 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353992 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353994 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353997 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.353999 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354001 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354004 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354006 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354009 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354011 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.356364 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354014 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354359 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354364 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354366 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354369 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354372 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354375 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354377 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354380 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354383 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354385 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354388 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354390 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354393 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354396 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354398 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354401 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354403 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354406 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354409 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.356848 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354412 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354414 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354417 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354419 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354422 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354424 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354427 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354429 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354432 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354434 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354437 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354439 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354442 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354445 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354449 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354451 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354454 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354457 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354459 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.357340 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354462 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354464 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354466 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354469 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354471 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354474 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354476 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354479 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354481 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354484 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354487 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354490 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354492 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354495 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354497 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354500 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354503 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354505 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354508 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354510 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.357787 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354512 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354515 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354517 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354519 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354522 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354524 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354529 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354531 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354534 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354536 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354539 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354544 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354547 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354550 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354553 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354556 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354560 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354563 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354566 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.358281 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354569 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354572 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354574 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354578 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354580 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354583 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354586 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354588 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354591 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354594 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354596 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354599 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354601 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354603 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354606 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354608 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354610 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354613 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354615 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354619 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.358758 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354622 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.354624 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354687 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354697 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354702 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354708 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354711 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354714 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354718 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354724 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354727 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354730 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354733 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354736 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354739 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354742 2570 flags.go:64] FLAG: --cgroup-root="" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354745 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354748 2570 flags.go:64] FLAG: --client-ca-file="" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354751 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354754 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354758 2570 flags.go:64] FLAG: --cluster-domain="" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354761 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354764 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354767 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 22:56:52.359236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354770 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354773 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354776 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354779 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354782 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354785 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354788 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354792 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354795 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354798 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354801 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354804 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354807 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354810 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354813 2570 flags.go:64] FLAG: --enable-server="true" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354816 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354833 2570 flags.go:64] FLAG: --event-burst="100" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354836 2570 flags.go:64] FLAG: --event-qps="50" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354839 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354842 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354845 2570 flags.go:64] FLAG: --eviction-hard="" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354849 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354851 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354854 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354857 2570 flags.go:64] FLAG: --eviction-soft="" Jul 31 22:56:52.359833 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354860 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354863 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354866 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354868 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354871 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354874 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354877 2570 flags.go:64] FLAG: --feature-gates="" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354880 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354883 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354886 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354889 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354891 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354894 2570 flags.go:64] FLAG: --help="false" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354897 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354901 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354904 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354907 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354910 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354913 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354916 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354918 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354921 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354925 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 22:56:52.360414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354927 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354930 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354933 2570 flags.go:64] FLAG: --kube-reserved="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354936 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354938 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354941 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354944 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354947 2570 flags.go:64] FLAG: --lock-file="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354950 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354952 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354955 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354960 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354963 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354965 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354968 2570 flags.go:64] FLAG: --logging-format="text" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354971 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354974 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354977 2570 flags.go:64] FLAG: --manifest-url="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354979 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354983 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354986 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354990 2570 flags.go:64] FLAG: --max-pods="110" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354992 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354996 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.354999 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 22:56:52.361035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355002 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355005 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355007 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355010 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355017 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355020 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355023 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355027 2570 flags.go:64] FLAG: --pod-cidr="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355029 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355034 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355037 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355040 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355043 2570 flags.go:64] FLAG: --port="10250" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355046 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355050 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06c65c10e8212ae8e" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355053 2570 flags.go:64] FLAG: --qos-reserved="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355056 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355059 2570 flags.go:64] FLAG: --register-node="true" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355062 2570 flags.go:64] FLAG: --register-with-taints="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355065 2570 flags.go:64] FLAG: --registry-burst="10" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355068 2570 flags.go:64] FLAG: --registry-qps="5" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355071 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355074 2570 flags.go:64] FLAG: --reserved-memory="" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355077 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355080 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 22:56:52.362084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355083 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355086 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355088 2570 flags.go:64] FLAG: --runonce="false" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355091 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355094 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355098 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355101 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355104 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355107 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355110 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355112 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355115 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355118 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355121 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355123 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355127 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355130 2570 flags.go:64] FLAG: --system-cgroups="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355133 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355137 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355140 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355143 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355148 2570 flags.go:64] FLAG: --tls-min-version="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355151 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355153 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355156 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 22:56:52.363064 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355159 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355162 2570 flags.go:64] FLAG: --v="2" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355166 2570 flags.go:64] FLAG: --version="false" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355169 2570 flags.go:64] FLAG: --vmodule="" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355172 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.355175 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355279 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355282 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355285 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355287 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355290 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355292 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355296 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355299 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355301 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355304 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355306 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355309 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355311 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355314 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355316 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.363705 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355319 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355321 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355327 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355329 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355331 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355335 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355339 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355343 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355346 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355348 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355351 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355353 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355356 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355358 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355362 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355364 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355366 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355369 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355372 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.364256 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355374 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355377 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355379 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355382 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355390 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355393 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355395 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355398 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355401 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355405 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355408 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355411 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355414 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355416 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355419 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355422 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355424 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355427 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355429 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.364688 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355432 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355435 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355438 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355441 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355443 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355445 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355448 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355450 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355454 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355456 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355459 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355461 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355464 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355467 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355469 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355472 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355474 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355477 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355480 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355482 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.365156 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355485 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355487 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355490 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355492 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355495 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355497 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355500 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355502 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355504 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355507 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355510 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355512 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355515 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355519 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355521 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355524 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355526 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355529 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355531 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355534 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.365627 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355536 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355539 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355542 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355544 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355547 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.355549 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.356189 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.363658 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.363674 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.364478 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.364487 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364541 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364546 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364549 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364553 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.366114 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364556 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364559 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364562 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364564 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364567 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364570 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364572 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364575 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364578 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364580 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364583 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364585 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364588 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364590 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364593 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364595 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364598 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364600 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364603 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364605 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.366480 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364608 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364611 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364614 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364618 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364622 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364626 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364630 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364633 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364636 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364639 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364641 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364644 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364647 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364649 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364652 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364654 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364657 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364659 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364662 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.366960 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364664 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364667 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364669 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364672 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364674 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364678 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364682 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364685 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364687 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364690 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364694 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364696 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364699 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364702 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364705 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364707 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364710 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364713 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364716 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.367418 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364719 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364721 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364724 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364727 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364729 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364732 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364734 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364737 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364739 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364742 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364744 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364747 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364749 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364752 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364755 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364757 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364760 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364763 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364766 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364769 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.367878 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364772 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364774 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364777 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364779 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364782 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364784 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364787 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364789 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364792 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364795 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364798 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364800 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364803 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364805 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364808 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364811 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.368344 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364814 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.364833 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364951 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364956 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364959 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364962 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364965 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364968 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364971 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364974 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364977 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364980 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364982 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364985 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.364988 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 22:56:52.368722 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365007 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365011 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365014 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365017 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365020 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365023 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365026 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365028 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365031 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365034 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365037 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365039 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365042 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365045 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365048 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365051 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365055 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365059 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365062 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365065 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 22:56:52.369193 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365067 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365070 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365072 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365074 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365077 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365081 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365084 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365087 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365090 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365093 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365095 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365099 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365102 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365106 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365108 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365111 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365114 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365117 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365120 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 22:56:52.369657 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365123 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365126 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365129 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365132 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365135 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365137 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365140 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365142 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365144 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365147 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365149 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365152 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365154 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365157 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365159 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365162 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365165 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365167 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365170 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 22:56:52.370108 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365172 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365175 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365178 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365180 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365183 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365185 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365187 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365190 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365192 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365195 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365197 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365200 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365202 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365205 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365207 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365210 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365212 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365215 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365218 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 22:56:52.370561 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365221 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365223 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365226 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365228 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365231 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365233 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365236 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365238 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:52.365241 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.365246 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.365361 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.368843 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.370369 2570 server.go:1019] "Starting client certificate rotation" Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.370466 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 22:56:52.371025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.370496 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 22:56:52.374483 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.374470 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 22:56:52.377798 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.377785 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 22:56:52.387727 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.387708 2570 log.go:25] "Validated CRI v1 runtime API" Jul 31 22:56:52.391293 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.391277 2570 log.go:25] "Validated CRI v1 image API" Jul 31 22:56:52.392097 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.392075 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 22:56:52.395417 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.395403 2570 fs.go:135] Filesystem UUIDs: map[214c2bdb-15d5-41b8-8c9d-82bcd62e4fcd:/dev/nvme0n1p3 2d85e837-a832-4d6b-aa31-1e40746930c8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 31 22:56:52.395472 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.395418 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 22:56:52.401202 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.401095 2570 manager.go:216] Machine: {Timestamp:2026-07-31 22:56:52.399967928 +0000 UTC m=+0.322235690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200872 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246d887e8c94448a3da8546154441b SystemUUID:ec246d88-7e8c-9444-8a3d-a8546154441b BootID:673464f3-9fb1-420d-891d-7163d124f458 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dc:36:b8:24:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dc:36:b8:24:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:ea:da:7e:db:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 22:56:52.401202 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.401193 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 22:56:52.401318 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.401259 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 22:56:52.403229 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.403214 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 22:56:52.404338 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.404319 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 22:56:52.404458 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.404342 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 22:56:52.405013 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.405003 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 22:56:52.405050 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.405015 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 22:56:52.405050 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.405022 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 22:56:52.405812 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.405801 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 22:56:52.406561 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.406551 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 22:56:52.410514 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410503 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 31 22:56:52.410629 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410619 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 22:56:52.410668 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410648 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 31 22:56:52.410668 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410657 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 22:56:52.410755 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410673 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 22:56:52.410755 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410682 2570 kubelet.go:398] "Adding apiserver pod source" Jul 31 22:56:52.410755 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.410690 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 22:56:52.411203 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.411190 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 22:56:52.411252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.411215 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 22:56:52.414589 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.414571 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 22:56:52.416039 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.416026 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 22:56:52.416083 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.416055 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 22:56:52.417287 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417275 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417290 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417296 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417301 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417309 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417314 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417322 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417327 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417333 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417338 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417346 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 22:56:52.417354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.417351 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 22:56:52.418482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.418472 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 22:56:52.418517 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.418483 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 22:56:52.421373 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.421362 2570 server.go:1268] "Started kubelet" Jul 31 22:56:52.421456 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.421425 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 22:56:52.421544 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.421496 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 22:56:52.421617 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.421603 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 22:56:52.422168 ip-10-0-137-58 systemd[1]: Started Kubernetes Kubelet. Jul 31 22:56:52.422995 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.422710 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 22:56:52.423580 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.423552 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 22:56:52.423705 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.423680 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 22:56:52.423790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.423716 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 22:56:52.423889 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.423875 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 31 22:56:52.428753 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.428737 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 22:56:52.428987 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.428975 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 22:56:52.429043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429031 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 22:56:52.429199 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.429179 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.429728 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429713 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 22:56:52.429832 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429732 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 22:56:52.429900 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429870 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 31 22:56:52.429900 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429878 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 31 22:56:52.430133 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430118 2570 factory.go:55] Registering systemd factory Jul 31 22:56:52.430222 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430136 2570 factory.go:223] Registration of the systemd container factory successfully Jul 31 22:56:52.430301 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.429051 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-58.ec2.internal.18c781e8661e20c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-58.ec2.internal,UID:ip-10-0-137-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-58.ec2.internal,},FirstTimestamp:2026-07-31 22:56:52.42133728 +0000 UTC m=+0.343605041,LastTimestamp:2026-07-31 22:56:52.42133728 +0000 UTC m=+0.343605041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-58.ec2.internal,}" Jul 31 22:56:52.431002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430561 2570 factory.go:153] Registering CRI-O factory Jul 31 22:56:52.431002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430578 2570 factory.go:223] Registration of the crio container factory successfully Jul 31 22:56:52.431002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430616 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 22:56:52.431002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430637 2570 factory.go:103] Registering Raw factory Jul 31 22:56:52.431002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.430649 2570 manager.go:1196] Started watching for new ooms in manager Jul 31 22:56:52.431252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.431023 2570 manager.go:349] Starting recovery of all containers Jul 31 22:56:52.431252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.429723 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 22:56:52.438165 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.438130 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 31 22:56:52.438309 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.438274 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 22:56:52.441316 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.441299 2570 manager.go:354] Recovery completed Jul 31 22:56:52.445681 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.445668 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.447419 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.447402 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.447489 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.447430 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.447489 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.447465 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.448044 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.448026 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.448134 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.448026 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 22:56:52.448134 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.448058 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 22:56:52.448134 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.448077 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 31 22:56:52.449257 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.449244 2570 policy_none.go:49] "None policy: Start" Jul 31 22:56:52.449321 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.449259 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 22:56:52.449321 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.449272 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 22:56:52.450043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.449982 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.450043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.450023 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.450043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.450037 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.450325 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.449993 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-58.ec2.internal.18c781e867ac177a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-58.ec2.internal,UID:ip-10-0-137-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-58.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-58.ec2.internal,},FirstTimestamp:2026-07-31 22:56:52.447418234 +0000 UTC m=+0.369685995,LastTimestamp:2026-07-31 22:56:52.447418234 +0000 UTC m=+0.369685995,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-58.ec2.internal,}" Jul 31 22:56:52.450385 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.450348 2570 policy_none.go:47] "Start" Jul 31 22:56:52.455164 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.455146 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rkgh" Jul 31 22:56:52.460064 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.460007 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-58.ec2.internal.18c781e867ac5893 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-58.ec2.internal,UID:ip-10-0-137-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-58.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-58.ec2.internal,},FirstTimestamp:2026-07-31 22:56:52.447434899 +0000 UTC m=+0.369702660,LastTimestamp:2026-07-31 22:56:52.447434899 +0000 UTC m=+0.369702660,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-58.ec2.internal,}" Jul 31 22:56:52.463236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.463219 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rkgh" Jul 31 22:56:52.475413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475395 2570 manager.go:337] "Starting Device Plugin manager" Jul 31 22:56:52.475505 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.475429 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 22:56:52.475505 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475441 2570 server.go:87] "Starting device plugin registration server" Jul 31 22:56:52.475907 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475673 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 22:56:52.475907 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475687 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 22:56:52.475907 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475749 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 22:56:52.475907 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475839 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 22:56:52.475907 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.475848 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 22:56:52.476233 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.476213 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 22:56:52.476319 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.476265 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.530943 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.530920 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 22:56:52.532068 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.532049 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 22:56:52.532132 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.532092 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 22:56:52.532132 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.532113 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 22:56:52.532234 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.532145 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 22:56:52.534548 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.534534 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:52.576505 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.576492 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.577213 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.577200 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.577287 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.577225 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.577287 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.577234 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.577287 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.577257 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.589166 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.589154 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.589215 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.589172 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-58.ec2.internal\": node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.610986 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.610941 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.633247 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.633228 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal"] Jul 31 22:56:52.633307 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.633274 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.633979 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.633966 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.634045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.633992 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.634045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.634003 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.635042 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635030 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.635156 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635142 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.635196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635170 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.635695 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635681 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.635790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635707 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.635790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635687 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.635790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635741 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.635790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635756 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.635790 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.635717 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.636968 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.636953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.637047 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.636982 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 22:56:52.637617 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.637601 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientMemory" Jul 31 22:56:52.637696 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.637629 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 22:56:52.637696 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.637642 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeHasSufficientPID" Jul 31 22:56:52.671399 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.671378 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-58.ec2.internal\" not found" node="ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.675515 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.675499 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-58.ec2.internal\" not found" node="ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.711612 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.711593 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.812092 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.812074 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.832451 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.832431 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6ebc48e958d442fefbec257d10b60ea-config\") pod \"kube-apiserver-proxy-ip-10-0-137-58.ec2.internal\" (UID: \"a6ebc48e958d442fefbec257d10b60ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.832525 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.832459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.832525 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.832484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.912733 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:52.912689 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:52.933101 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6ebc48e958d442fefbec257d10b60ea-config\") pod \"kube-apiserver-proxy-ip-10-0-137-58.ec2.internal\" (UID: \"a6ebc48e958d442fefbec257d10b60ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.933196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.933196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.933196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.933196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6ebc48e958d442fefbec257d10b60ea-config\") pod \"kube-apiserver-proxy-ip-10-0-137-58.ec2.internal\" (UID: \"a6ebc48e958d442fefbec257d10b60ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.933196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.933183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6acb4bf471420828b96cf11e2a7009d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal\" (UID: \"6acb4bf471420828b96cf11e2a7009d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.974252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.974220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:52.978949 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:52.978723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:53.013430 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.013406 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:53.114033 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.114010 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:53.214596 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.214546 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:53.289334 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.289314 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:53.315133 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.315106 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-58.ec2.internal\" not found" Jul 31 22:56:53.330774 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.330756 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:53.370726 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.370707 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 22:56:53.371115 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.370822 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 22:56:53.371115 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.370872 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 22:56:53.371115 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.370880 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 22:56:53.412614 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.412593 2570 apiserver.go:52] "Watching apiserver" Jul 31 22:56:53.429437 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.429406 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 22:56:53.429437 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.429427 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" Jul 31 22:56:53.435306 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.435285 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 22:56:53.435648 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.435628 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vmspv","openshift-image-registry/node-ca-2rfbw","openshift-multus/multus-additional-cni-plugins-2rcc8","openshift-multus/multus-dk5cv","openshift-ovn-kubernetes/ovnkube-node-nv77q","kube-system/global-pull-secret-syncer-2rrsk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq","openshift-cluster-node-tuning-operator/tuned-fp4tg","openshift-dns/node-resolver-xzgjv","openshift-multus/network-metrics-daemon-9ftrs","openshift-network-diagnostics/network-check-target-qvnqs","openshift-network-operator/iptables-alerter-7jdfm"] Jul 31 22:56:53.437147 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.437131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.438240 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.438218 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.439600 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.439423 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 22:56:53.439600 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.439519 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" Jul 31 22:56:53.440096 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.440077 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.440651 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.440621 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g8hxx\"" Jul 31 22:56:53.441491 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.441280 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 22:56:53.441491 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.441425 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 22:56:53.441696 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.441675 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.441696 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.441689 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 22:56:53.441812 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.441701 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.443610 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.443591 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.444912 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.444872 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.445033 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.444944 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:56:53.445033 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.445016 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.445145 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.445074 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 22:56:53.445145 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.445081 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 22:56:53.446188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.446170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.447461 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.447443 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.448863 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.448821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.450099 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450079 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.450208 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.450151 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:56:53.450208 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450175 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 22:56:53.450307 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6g9hb\"" Jul 31 22:56:53.450494 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450477 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 22:56:53.450557 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450523 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zr898\"" Jul 31 22:56:53.450557 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450547 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.450673 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450479 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xvtzz\"" Jul 31 22:56:53.450673 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450551 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.450673 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450486 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-56w8s\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450918 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wmdp\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450946 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450955 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450862 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450986 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 22:56:53.451086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.450944 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f5q7j\"" Jul 31 22:56:53.451531 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451096 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.451531 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451154 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.451531 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451218 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 22:56:53.451531 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451492 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 22:56:53.451531 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451500 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.451775 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.451720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:53.451829 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.451777 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:56:53.453350 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.453333 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal"] Jul 31 22:56:53.453414 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.453401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.454440 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.454422 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.454495 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.454422 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.455109 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.455094 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwzbq\"" Jul 31 22:56:53.457910 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.457874 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 22:56:53.457910 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.457884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 22:56:53.458043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.457911 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-57n9m\"" Jul 31 22:56:53.458142 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.458126 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 22:56:53.461166 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.461151 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 22:56:53.463668 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.463654 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal"] Jul 31 22:56:53.466252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.466217 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 22:56:53.466486 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.466458 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 22:51:53 +0000 UTC" deadline="2028-01-23 14:19:24.301778375 +0000 UTC" Jul 31 22:56:53.466557 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.466486 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12975h22m30.835296331s" Jul 31 22:56:53.469145 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.469116 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:53.482654 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.482637 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2rpbq" Jul 31 22:56:53.495474 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.495456 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2rpbq" Jul 31 22:56:53.532534 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.532516 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 22:56:53.536572 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536551 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-kubelet\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.536681 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-var-lib-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.536681 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.536681 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-bin\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.536681 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0530a-f47c-4900-95e6-e9fec3c7924d-host\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dd0530a-f47c-4900-95e6-e9fec3c7924d-serviceca\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-socket-dir-parent\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-env-overrides\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-dbus\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.536880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536867 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cni-binary-copy\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536891 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.536978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-tmp-dir\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537002 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczqr\" (UniqueName: \"kubernetes.io/projected/832f6392-a691-4432-a15b-d120dfc94edc-kube-api-access-fczqr\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-system-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537050 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-etc-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cnibin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq65c\" (UniqueName: \"kubernetes.io/projected/842f999d-a623-4c09-8218-099a42fe84cc-kube-api-access-mq65c\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537153 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/832f6392-a691-4432-a15b-d120dfc94edc-iptables-alerter-script\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/832f6392-a691-4432-a15b-d120dfc94edc-host-slash\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537200 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglbk\" (UniqueName: \"kubernetes.io/projected/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-kube-api-access-hglbk\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537223 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-systemd-units\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537272 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-os-release\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537333 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-k8s-cni-cncf-io\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-hostroot\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysconfig\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpr6\" (UniqueName: \"kubernetes.io/projected/7dd0530a-f47c-4900-95e6-e9fec3c7924d-kube-api-access-9lpr6\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-multus\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-multus-certs\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537704 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-kubernetes\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537755 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-tuned\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:53.537827 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-hosts-file\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-bin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23ac3832-68eb-46ab-910a-d87d76403e76-ovn-node-metrics-cert\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-tmp\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537962 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs2q\" (UniqueName: \"kubernetes.io/projected/b5d34807-3975-4e79-b5d0-daeb9083eaf1-kube-api-access-dbs2q\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.537986 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-node-log\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-systemd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-ovn\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5hd\" (UniqueName: \"kubernetes.io/projected/1871d052-b938-4d33-9aa5-cdad64a19843-kube-api-access-4h5hd\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqwh\" (UniqueName: \"kubernetes.io/projected/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-kube-api-access-9rqwh\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-netns\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-slash\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-sys\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-conf-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-etc-kubernetes\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-log-socket\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.538413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-host\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-os-release\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-device-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-system-cni-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-cnibin\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-daemon-config\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-config\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-lib-modules\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-var-lib-kubelet\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-kubelet\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538619 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-script-lib\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tb76\" (UniqueName: \"kubernetes.io/projected/23ac3832-68eb-46ab-910a-d87d76403e76-kube-api-access-6tb76\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-netns\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-kubelet-config\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.539234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538717 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9z4\" (UniqueName: \"kubernetes.io/projected/c67ad40e-3221-4384-bbe2-a000c3b61847-kube-api-access-6w9z4\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-conf\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-systemd\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-run\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43cb94e5-0771-4262-bc66-258d46c52571-agent-certs\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43cb94e5-0771-4262-bc66-258d46c52571-konnectivity-ca\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-netd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.539731 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.538971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-modprobe-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.639328 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-os-release\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-device-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-system-cni-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-cnibin\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-cnibin\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-os-release\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-system-cni-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-device-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-daemon-config\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.639519 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-config\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-lib-modules\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-var-lib-kubelet\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-kubelet\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-script-lib\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-var-lib-kubelet\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639687 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-kubelet\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-lib-modules\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tb76\" (UniqueName: \"kubernetes.io/projected/23ac3832-68eb-46ab-910a-d87d76403e76-kube-api-access-6tb76\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-netns\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-kubelet-config\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9z4\" (UniqueName: \"kubernetes.io/projected/c67ad40e-3221-4384-bbe2-a000c3b61847-kube-api-access-6w9z4\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-netns\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639875 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-kubelet-config\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639876 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-conf\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-systemd\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-run\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640069 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.639986 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-conf\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-run\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-systemd\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43cb94e5-0771-4262-bc66-258d46c52571-agent-certs\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640082 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-config\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43cb94e5-0771-4262-bc66-258d46c52571-konnectivity-ca\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640187 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-netd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-ovnkube-script-lib\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640219 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-modprobe-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640238 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-daemon-config\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-kubelet\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-netd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-var-lib-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-kubelet\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640310 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-var-lib-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.640930 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640335 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-modprobe-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-run-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-bin\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640369 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-cni-bin\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0530a-f47c-4900-95e6-e9fec3c7924d-host\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dd0530a-f47c-4900-95e6-e9fec3c7924d-serviceca\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0530a-f47c-4900-95e6-e9fec3c7924d-host\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640461 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640484 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-socket-dir-parent\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-env-overrides\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-dbus\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cni-binary-copy\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640575 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-socket-dir-parent\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640621 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.641824 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640640 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43cb94e5-0771-4262-bc66-258d46c52571-konnectivity-ca\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640652 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/091a5632-061d-4b31-8e30-f03fffa6a68d-dbus\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640652 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-tmp-dir\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczqr\" (UniqueName: \"kubernetes.io/projected/832f6392-a691-4432-a15b-d120dfc94edc-kube-api-access-fczqr\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-system-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-etc-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.640768 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cnibin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq65c\" (UniqueName: \"kubernetes.io/projected/842f999d-a623-4c09-8218-099a42fe84cc-kube-api-access-mq65c\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.640836 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:56:54.140815898 +0000 UTC m=+2.063083672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/832f6392-a691-4432-a15b-d120dfc94edc-iptables-alerter-script\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/832f6392-a691-4432-a15b-d120dfc94edc-host-slash\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglbk\" (UniqueName: \"kubernetes.io/projected/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-kube-api-access-hglbk\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-systemd-units\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-os-release\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.642765 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-k8s-cni-cncf-io\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641103 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641120 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-etc-openvswitch\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-hostroot\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641130 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cni-binary-copy\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23ac3832-68eb-46ab-910a-d87d76403e76-env-overrides\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.640835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dd0530a-f47c-4900-95e6-e9fec3c7924d-serviceca\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-system-cni-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641109 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-systemd-units\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-registration-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-hostroot\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641214 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-cnibin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641235 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641237 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-tmp-dir\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641252 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-k8s-cni-cncf-io\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/832f6392-a691-4432-a15b-d120dfc94edc-host-slash\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.643638 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-os-release\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641386 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-sys-fs\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641422 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysconfig\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpr6\" (UniqueName: \"kubernetes.io/projected/7dd0530a-f47c-4900-95e6-e9fec3c7924d-kube-api-access-9lpr6\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641475 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysconfig\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-multus\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-multus-certs\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-sysctl-d\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641602 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-kubernetes\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-tuned\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-multus\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641634 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1871d052-b938-4d33-9aa5-cdad64a19843-cni-binary-copy\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641687 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-multus-certs\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.644496 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641737 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/832f6392-a691-4432-a15b-d120dfc94edc-iptables-alerter-script\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641742 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-kubernetes\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1871d052-b938-4d33-9aa5-cdad64a19843-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-etc-selinux\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-hosts-file\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-bin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641838 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-hosts-file\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23ac3832-68eb-46ab-910a-d87d76403e76-ovn-node-metrics-cert\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-tmp\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.641904 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs2q\" (UniqueName: \"kubernetes.io/projected/b5d34807-3975-4e79-b5d0-daeb9083eaf1-kube-api-access-dbs2q\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641900 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-var-lib-cni-bin\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-node-log\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.641969 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:56:54.141951961 +0000 UTC m=+2.064219726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.641979 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-node-log\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-systemd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-ovn\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5hd\" (UniqueName: \"kubernetes.io/projected/1871d052-b938-4d33-9aa5-cdad64a19843-kube-api-access-4h5hd\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqwh\" (UniqueName: \"kubernetes.io/projected/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-kube-api-access-9rqwh\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642096 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-systemd\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642100 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-run-ovn\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-netns\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642171 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-host-run-netns\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-slash\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-sys\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642322 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-host-slash\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-conf-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-sys\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-multus-conf-dir\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-etc-kubernetes\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-log-socket\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642411 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-etc-kubernetes\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.645512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23ac3832-68eb-46ab-910a-d87d76403e76-log-socket\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-host\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d34807-3975-4e79-b5d0-daeb9083eaf1-host\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.642651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c67ad40e-3221-4384-bbe2-a000c3b61847-socket-dir\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.643985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-tmp\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.644067 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43cb94e5-0771-4262-bc66-258d46c52571-agent-certs\") pod \"konnectivity-agent-vmspv\" (UID: \"43cb94e5-0771-4262-bc66-258d46c52571\") " pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.644267 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d34807-3975-4e79-b5d0-daeb9083eaf1-etc-tuned\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.646087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.644542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23ac3832-68eb-46ab-910a-d87d76403e76-ovn-node-metrics-cert\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.650292 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.650271 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:56:53.650427 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.650300 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:56:53.650427 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.650308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tb76\" (UniqueName: \"kubernetes.io/projected/23ac3832-68eb-46ab-910a-d87d76403e76-kube-api-access-6tb76\") pod \"ovnkube-node-nv77q\" (UID: \"23ac3832-68eb-46ab-910a-d87d76403e76\") " pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.650427 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.650315 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:53.650427 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.650388 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9z4\" (UniqueName: \"kubernetes.io/projected/c67ad40e-3221-4384-bbe2-a000c3b61847-kube-api-access-6w9z4\") pod \"aws-ebs-csi-driver-node-qlxtq\" (UID: \"c67ad40e-3221-4384-bbe2-a000c3b61847\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.650427 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.650426 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:56:54.150398361 +0000 UTC m=+2.072666119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:53.651530 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.651475 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczqr\" (UniqueName: \"kubernetes.io/projected/832f6392-a691-4432-a15b-d120dfc94edc-kube-api-access-fczqr\") pod \"iptables-alerter-7jdfm\" (UID: \"832f6392-a691-4432-a15b-d120dfc94edc\") " pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.652864 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.652840 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5hd\" (UniqueName: \"kubernetes.io/projected/1871d052-b938-4d33-9aa5-cdad64a19843-kube-api-access-4h5hd\") pod \"multus-additional-cni-plugins-2rcc8\" (UID: \"1871d052-b938-4d33-9aa5-cdad64a19843\") " pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.653373 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.653342 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqwh\" (UniqueName: \"kubernetes.io/projected/9fcb9bc0-23e8-4f43-9468-54fb1c39bd23-kube-api-access-9rqwh\") pod \"node-resolver-xzgjv\" (UID: \"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23\") " pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.653461 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.653446 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq65c\" (UniqueName: \"kubernetes.io/projected/842f999d-a623-4c09-8218-099a42fe84cc-kube-api-access-mq65c\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:53.653631 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.653613 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglbk\" (UniqueName: \"kubernetes.io/projected/0cc84881-0fc3-410e-bb1d-fe610a4c85a6-kube-api-access-hglbk\") pod \"multus-dk5cv\" (UID: \"0cc84881-0fc3-410e-bb1d-fe610a4c85a6\") " pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.654345 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.654326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpr6\" (UniqueName: \"kubernetes.io/projected/7dd0530a-f47c-4900-95e6-e9fec3c7924d-kube-api-access-9lpr6\") pod \"node-ca-2rfbw\" (UID: \"7dd0530a-f47c-4900-95e6-e9fec3c7924d\") " pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.654463 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.654448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs2q\" (UniqueName: \"kubernetes.io/projected/b5d34807-3975-4e79-b5d0-daeb9083eaf1-kube-api-access-dbs2q\") pod \"tuned-fp4tg\" (UID: \"b5d34807-3975-4e79-b5d0-daeb9083eaf1\") " pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.710876 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.710844 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ebc48e958d442fefbec257d10b60ea.slice/crio-c0dc29ef210c8f47c3439dd91f75601f00ef8a483f2f4af2cf4b7c7843db6568 WatchSource:0}: Error finding container c0dc29ef210c8f47c3439dd91f75601f00ef8a483f2f4af2cf4b7c7843db6568: Status 404 returned error can't find the container with id c0dc29ef210c8f47c3439dd91f75601f00ef8a483f2f4af2cf4b7c7843db6568 Jul 31 22:56:53.711459 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.711440 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acb4bf471420828b96cf11e2a7009d4.slice/crio-f3e3a5b188b300f9ac1c1d95770403337bdd9103606cf497029e356945b2771f WatchSource:0}: Error finding container f3e3a5b188b300f9ac1c1d95770403337bdd9103606cf497029e356945b2771f: Status 404 returned error can't find the container with id f3e3a5b188b300f9ac1c1d95770403337bdd9103606cf497029e356945b2771f Jul 31 22:56:53.717100 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.717084 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 22:56:53.768706 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.768673 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:56:53.775179 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.775161 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rfbw" Jul 31 22:56:53.777679 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.777662 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb94e5_0771_4262_bc66_258d46c52571.slice/crio-83102c9abc0f5cd6ef2b40f202bd39e7e0ffb9bbc21f7cad7900cae3df7762da WatchSource:0}: Error finding container 83102c9abc0f5cd6ef2b40f202bd39e7e0ffb9bbc21f7cad7900cae3df7762da: Status 404 returned error can't find the container with id 83102c9abc0f5cd6ef2b40f202bd39e7e0ffb9bbc21f7cad7900cae3df7762da Jul 31 22:56:53.783271 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.783251 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd0530a_f47c_4900_95e6_e9fec3c7924d.slice/crio-8bbd9b75199a7dcf7290d0dac11b42da5acdbe39d8c36a042ca0e5be9196f5d8 WatchSource:0}: Error finding container 8bbd9b75199a7dcf7290d0dac11b42da5acdbe39d8c36a042ca0e5be9196f5d8: Status 404 returned error can't find the container with id 8bbd9b75199a7dcf7290d0dac11b42da5acdbe39d8c36a042ca0e5be9196f5d8 Jul 31 22:56:53.793330 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.793309 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" Jul 31 22:56:53.801796 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.801770 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1871d052_b938_4d33_9aa5_cdad64a19843.slice/crio-e51db2eac0edc229a6a012c057d3f4c22c8bca65892feede957145ed7e830213 WatchSource:0}: Error finding container e51db2eac0edc229a6a012c057d3f4c22c8bca65892feede957145ed7e830213: Status 404 returned error can't find the container with id e51db2eac0edc229a6a012c057d3f4c22c8bca65892feede957145ed7e830213 Jul 31 22:56:53.805969 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.805954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk5cv" Jul 31 22:56:53.813605 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.813588 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc84881_0fc3_410e_bb1d_fe610a4c85a6.slice/crio-f5a604bc9b3d5696b43e7812aec8da4f38144f2816f2d56e8fd4553be47ff011 WatchSource:0}: Error finding container f5a604bc9b3d5696b43e7812aec8da4f38144f2816f2d56e8fd4553be47ff011: Status 404 returned error can't find the container with id f5a604bc9b3d5696b43e7812aec8da4f38144f2816f2d56e8fd4553be47ff011 Jul 31 22:56:53.824472 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.824456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:56:53.831882 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.831865 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ac3832_68eb_46ab_910a_d87d76403e76.slice/crio-e72422ef4acf18e5ae5bf46fca58a1d131d292d4da73c9b192953b825e76f498 WatchSource:0}: Error finding container e72422ef4acf18e5ae5bf46fca58a1d131d292d4da73c9b192953b825e76f498: Status 404 returned error can't find the container with id e72422ef4acf18e5ae5bf46fca58a1d131d292d4da73c9b192953b825e76f498 Jul 31 22:56:53.837061 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.837044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" Jul 31 22:56:53.845035 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.845016 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67ad40e_3221_4384_bbe2_a000c3b61847.slice/crio-6b352e5a838585edfc063ea455600f587d87ac9a07e3e0a9748d8472010329da WatchSource:0}: Error finding container 6b352e5a838585edfc063ea455600f587d87ac9a07e3e0a9748d8472010329da: Status 404 returned error can't find the container with id 6b352e5a838585edfc063ea455600f587d87ac9a07e3e0a9748d8472010329da Jul 31 22:56:53.855054 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.855040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" Jul 31 22:56:53.859983 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.859971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzgjv" Jul 31 22:56:53.862802 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.862741 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d34807_3975_4e79_b5d0_daeb9083eaf1.slice/crio-ab000b0b42dc1a55c721bc35cb063a629194bf538ca9461f9fcd5e248c78e198 WatchSource:0}: Error finding container ab000b0b42dc1a55c721bc35cb063a629194bf538ca9461f9fcd5e248c78e198: Status 404 returned error can't find the container with id ab000b0b42dc1a55c721bc35cb063a629194bf538ca9461f9fcd5e248c78e198 Jul 31 22:56:53.869136 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.869116 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb9bc0_23e8_4f43_9468_54fb1c39bd23.slice/crio-34f82135e68ffe3c99f3cdb8ce954de07f5e70a40bb92bba7cc69b2622b43fdd WatchSource:0}: Error finding container 34f82135e68ffe3c99f3cdb8ce954de07f5e70a40bb92bba7cc69b2622b43fdd: Status 404 returned error can't find the container with id 34f82135e68ffe3c99f3cdb8ce954de07f5e70a40bb92bba7cc69b2622b43fdd Jul 31 22:56:53.881319 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:53.881301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jdfm" Jul 31 22:56:53.890896 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:56:53.890878 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832f6392_a691_4432_a15b_d120dfc94edc.slice/crio-644a539edf9727126fb500b40ce787f22c787f536f8992b4fe32c048fa73e363 WatchSource:0}: Error finding container 644a539edf9727126fb500b40ce787f22c787f536f8992b4fe32c048fa73e363: Status 404 returned error can't find the container with id 644a539edf9727126fb500b40ce787f22c787f536f8992b4fe32c048fa73e363 Jul 31 22:56:53.892376 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.892347 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-7jdfm_openshift-network-operator(832f6392-a691-4432-a15b-d120dfc94edc): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 31 22:56:53.892434 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:53.892394 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-7jdfm" podUID="832f6392-a691-4432-a15b-d120dfc94edc" Jul 31 22:56:54.147362 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.147297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:54.147362 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.147346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:54.147504 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.147427 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:54.147504 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.147447 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:54.147504 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.147473 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:56:55.147459132 +0000 UTC m=+3.069726881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:54.147504 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.147497 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:56:55.147484182 +0000 UTC m=+3.069751931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:54.247898 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.247868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:54.248038 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.247995 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:56:54.248038 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.248014 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:56:54.248038 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.248026 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:54.248177 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.248072 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:56:55.248059346 +0000 UTC m=+3.170327095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:54.497807 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.497629 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 22:51:54 +0000 UTC" deadline="2028-02-16 07:20:25.927516662 +0000 UTC" Jul 31 22:56:54.497807 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.497702 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13544h23m31.429819707s" Jul 31 22:56:54.563486 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.563423 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" event={"ID":"a6ebc48e958d442fefbec257d10b60ea","Type":"ContainerStarted","Data":"c0dc29ef210c8f47c3439dd91f75601f00ef8a483f2f4af2cf4b7c7843db6568"} Jul 31 22:56:54.571553 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.571515 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" event={"ID":"6acb4bf471420828b96cf11e2a7009d4","Type":"ContainerStarted","Data":"f3e3a5b188b300f9ac1c1d95770403337bdd9103606cf497029e356945b2771f"} Jul 31 22:56:54.577958 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.577441 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jdfm" event={"ID":"832f6392-a691-4432-a15b-d120dfc94edc","Type":"ContainerStarted","Data":"644a539edf9727126fb500b40ce787f22c787f536f8992b4fe32c048fa73e363"} Jul 31 22:56:54.583828 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:54.583751 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:74b8d9cff26ab06c9d7bbeb115f3d2d4319dafb5e9f367b231d40f610b152391\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-7jdfm" podUID="832f6392-a691-4432-a15b-d120dfc94edc" Jul 31 22:56:54.594801 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.594777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" event={"ID":"c67ad40e-3221-4384-bbe2-a000c3b61847","Type":"ContainerStarted","Data":"6b352e5a838585edfc063ea455600f587d87ac9a07e3e0a9748d8472010329da"} Jul 31 22:56:54.600634 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.600610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk5cv" event={"ID":"0cc84881-0fc3-410e-bb1d-fe610a4c85a6","Type":"ContainerStarted","Data":"f5a604bc9b3d5696b43e7812aec8da4f38144f2816f2d56e8fd4553be47ff011"} Jul 31 22:56:54.608112 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.608087 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rfbw" event={"ID":"7dd0530a-f47c-4900-95e6-e9fec3c7924d","Type":"ContainerStarted","Data":"8bbd9b75199a7dcf7290d0dac11b42da5acdbe39d8c36a042ca0e5be9196f5d8"} Jul 31 22:56:54.619913 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.619880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vmspv" event={"ID":"43cb94e5-0771-4262-bc66-258d46c52571","Type":"ContainerStarted","Data":"83102c9abc0f5cd6ef2b40f202bd39e7e0ffb9bbc21f7cad7900cae3df7762da"} Jul 31 22:56:54.639224 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.639045 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzgjv" event={"ID":"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23","Type":"ContainerStarted","Data":"34f82135e68ffe3c99f3cdb8ce954de07f5e70a40bb92bba7cc69b2622b43fdd"} Jul 31 22:56:54.657412 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.657302 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" event={"ID":"b5d34807-3975-4e79-b5d0-daeb9083eaf1","Type":"ContainerStarted","Data":"ab000b0b42dc1a55c721bc35cb063a629194bf538ca9461f9fcd5e248c78e198"} Jul 31 22:56:54.669842 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.669802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"e72422ef4acf18e5ae5bf46fca58a1d131d292d4da73c9b192953b825e76f498"} Jul 31 22:56:54.695389 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.695362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerStarted","Data":"e51db2eac0edc229a6a012c057d3f4c22c8bca65892feede957145ed7e830213"} Jul 31 22:56:54.708675 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:54.708628 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:55.154757 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.154720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:55.154976 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.154800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:55.154976 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.154956 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:55.155115 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.155030 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:56:57.155010029 +0000 UTC m=+5.077277781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:55.155452 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.155431 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:55.155543 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.155490 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:56:57.155474904 +0000 UTC m=+5.077742658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:55.255993 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.255731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:55.255993 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.255951 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:56:55.255993 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.255977 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:56:55.255993 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.255991 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:55.256336 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.256047 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:56:57.256028353 +0000 UTC m=+5.178296121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:55.498771 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.498645 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 22:51:54 +0000 UTC" deadline="2028-05-12 13:02:47.202358932 +0000 UTC" Jul 31 22:56:55.498771 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.498683 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15614h5m51.703680444s" Jul 31 22:56:55.533316 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.533278 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:55.533457 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.533410 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:56:55.533878 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.533816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:55.534008 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.533948 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:56:55.534065 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.534018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:55.534133 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:55.534097 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:56:55.956145 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:55.956071 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:56.424344 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:56.424165 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 22:56:57.170743 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.170703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:57.171365 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.170794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:57.171365 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.170957 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:57.171365 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.171030 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:57:01.171008141 +0000 UTC m=+9.093275899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:56:57.171601 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.171451 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:57.171601 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.171510 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:57:01.17149309 +0000 UTC m=+9.093760854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:56:57.272046 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.271798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:57.272046 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.271986 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:56:57.272046 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.272007 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:56:57.272046 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.272021 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:57.272374 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.272080 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:57:01.272063167 +0000 UTC m=+9.194330932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:56:57.533132 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.533044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:57.533689 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.533653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:56:57.538338 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.538135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:57.538338 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.538293 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:56:57.538517 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:57.538371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:57.538574 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:57.538522 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:56:59.533231 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:59.533196 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:56:59.533722 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:59.533294 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:56:59.533722 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:56:59.533312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:56:59.533722 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:59.533438 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:56:59.533722 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:59.533592 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:56:59.533970 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:56:59.533766 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:01.207091 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.207061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:01.207577 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.207127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:01.207577 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.207255 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:01.207577 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.207328 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:57:09.207306463 +0000 UTC m=+17.129574229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:01.207577 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.207247 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:01.207577 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.207395 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:57:09.207382604 +0000 UTC m=+17.129650358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:01.308532 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.308504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:01.308722 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.308654 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:57:01.308722 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.308676 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:57:01.308722 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.308688 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:01.308905 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.308741 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:57:09.308723747 +0000 UTC m=+17.230991498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:01.533065 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.532970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:01.533065 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.533017 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:01.533290 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.533107 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:01.533290 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.533160 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:01.533290 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:01.533219 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:01.533437 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:01.533304 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:03.533162 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:03.533087 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:03.533639 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:03.533082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:03.533639 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:03.533218 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:03.533639 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:03.533275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:03.533639 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:03.533409 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:03.533639 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:03.533502 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:05.532533 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:05.532501 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:05.532533 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:05.532501 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:05.532980 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:05.532780 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:05.532980 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:05.532886 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:05.533072 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:05.532978 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:05.533107 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:05.533069 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:07.532625 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:07.532586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:07.532625 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:07.532608 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:07.533087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:07.532610 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:07.533087 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:07.532713 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:07.533087 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:07.532777 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:07.533087 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:07.532877 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:09.278916 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.278869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:09.279405 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.278954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:09.279405 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.279071 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:09.279405 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.279089 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:09.279405 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.279146 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:57:25.279123829 +0000 UTC m=+33.201391580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:09.279405 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.279166 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:57:25.279155896 +0000 UTC m=+33.201423644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:09.380173 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.380146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:09.380334 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.380318 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:57:09.380398 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.380337 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:57:09.380398 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.380348 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:09.380490 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.380400 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:57:25.380387676 +0000 UTC m=+33.302655428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:09.533090 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.533027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:09.533090 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.533046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:09.533237 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:09.533093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:09.533237 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.533111 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:09.533237 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.533189 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:09.533316 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:09.533260 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:11.533049 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:11.533020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:11.533049 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:11.533041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:11.533505 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:11.533128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:11.533505 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:11.533134 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:11.533505 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:11.533220 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:11.533505 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:11.533298 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:13.532429 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.532298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:13.532703 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.532330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:13.532703 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:13.532527 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:13.532703 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.532359 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:13.532703 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:13.532667 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:13.532875 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:13.532749 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:13.758157 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.757583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk5cv" event={"ID":"0cc84881-0fc3-410e-bb1d-fe610a4c85a6","Type":"ContainerStarted","Data":"e5848b2d502edaf231ff7a67ddeb9d9105846013b245e7f715bb7ae52727525d"} Jul 31 22:57:13.760717 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.760674 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" event={"ID":"b5d34807-3975-4e79-b5d0-daeb9083eaf1","Type":"ContainerStarted","Data":"48325996b817e25c7611f8581960a2133357c3c6fcc83d8fc30152aa7ca2c93a"} Jul 31 22:57:13.763270 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.763253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 22:57:13.763623 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.763605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"8514eb2bd7b4cdf1fba5945c47b4341c8d3cfb8a3bed1303586760c48bf4c09d"} Jul 31 22:57:13.763710 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.763674 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"905c89f1860b60899014f57058ca83b3d1f1a9c0f41eeca7ea34b5ea2acba4c6"} Jul 31 22:57:13.768489 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.768464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" event={"ID":"a6ebc48e958d442fefbec257d10b60ea","Type":"ContainerStarted","Data":"14b5d731504566d8a350b2a18300939c920fca74482e96f6e69e1f66f0b05f16"} Jul 31 22:57:13.780451 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.780397 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dk5cv" podStartSLOduration=0.929529838 podStartE2EDuration="20.780378028s" totalImagesPullingTime="19.85084819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:13.779633226 +0000 UTC m=+21.701900996" watchObservedRunningTime="2026-07-31 22:57:13.780378028 +0000 UTC m=+21.702645799" Jul 31 22:57:13.795979 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:13.795733 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-58.ec2.internal" podStartSLOduration=19.795713603 podStartE2EDuration="19.795713603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:13.795426266 +0000 UTC m=+21.717694033" watchObservedRunningTime="2026-07-31 22:57:13.795713603 +0000 UTC m=+21.717981397" Jul 31 22:57:14.772319 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.772091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" event={"ID":"c67ad40e-3221-4384-bbe2-a000c3b61847","Type":"ContainerStarted","Data":"9be9827eb570c44cf2a95a4dfcb506e94b89d32a5d68cdfaf59e4c83e9366ff4"} Jul 31 22:57:14.773512 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.773487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rfbw" event={"ID":"7dd0530a-f47c-4900-95e6-e9fec3c7924d","Type":"ContainerStarted","Data":"c0e47f1894a881d956b0d013a678832cf32333ba4eb89eaa10b8f784714653f6"} Jul 31 22:57:14.774721 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.774686 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vmspv" event={"ID":"43cb94e5-0771-4262-bc66-258d46c52571","Type":"ContainerStarted","Data":"6e08c5df20e736a2d1d10fcd424f51900eabf5fc4233726979a6ee7be384ffd1"} Jul 31 22:57:14.776020 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.775996 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzgjv" event={"ID":"9fcb9bc0-23e8-4f43-9468-54fb1c39bd23","Type":"ContainerStarted","Data":"ddc4ad17fc6889cafebef42a25b500bc3e356f3564aa173740f195583767d79f"} Jul 31 22:57:14.778550 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.778533 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 22:57:14.778860 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.778844 2570 generic.go:356] "Generic (PLEG): container finished" podID="23ac3832-68eb-46ab-910a-d87d76403e76" containerID="8514eb2bd7b4cdf1fba5945c47b4341c8d3cfb8a3bed1303586760c48bf4c09d" exitCode=1 Jul 31 22:57:14.778926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.778900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerDied","Data":"8514eb2bd7b4cdf1fba5945c47b4341c8d3cfb8a3bed1303586760c48bf4c09d"} Jul 31 22:57:14.778969 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.778949 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"8f566e41754d44a3b099156bfc488fbfa641460e58f32121bb857336dd1f287d"} Jul 31 22:57:14.779005 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.778986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"c939cb0fd6db20cb6935951becf5b6a0b24bf40153fda3584e2612ead75a5ac2"} Jul 31 22:57:14.779045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.779021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"4a79b215cc8aadc4a4c8a378dc65bbff569f2daaafeb1a98d6f87a7ee1f3cdf4"} Jul 31 22:57:14.779094 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.779047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"bd67791bd08f699a7573b9bf5de02d5d8a963400c34edef7485be74bac1b8053"} Jul 31 22:57:14.780212 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.780192 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="afcc0737c08ca311099aef3fbada248b31d3cf3842d796495ebc41ca0d88df3d" exitCode=0 Jul 31 22:57:14.780312 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.780252 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"afcc0737c08ca311099aef3fbada248b31d3cf3842d796495ebc41ca0d88df3d"} Jul 31 22:57:14.781707 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.781688 2570 generic.go:356] "Generic (PLEG): container finished" podID="6acb4bf471420828b96cf11e2a7009d4" containerID="7ffd75bea1bec449b1f96fc57a6f610761b3d40a089d0f478f9f037f9c84ed64" exitCode=0 Jul 31 22:57:14.781801 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.781773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" event={"ID":"6acb4bf471420828b96cf11e2a7009d4","Type":"ContainerDied","Data":"7ffd75bea1bec449b1f96fc57a6f610761b3d40a089d0f478f9f037f9c84ed64"} Jul 31 22:57:14.793304 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.793270 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fp4tg" podStartSLOduration=2.236282504 podStartE2EDuration="21.793257171s" totalImagesPullingTime="19.556974667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:13.816417796 +0000 UTC m=+21.738685567" watchObservedRunningTime="2026-07-31 22:57:14.793257171 +0000 UTC m=+22.715524941" Jul 31 22:57:14.810848 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.810794 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2rfbw" podStartSLOduration=2.177317369 podStartE2EDuration="21.810782872s" totalImagesPullingTime="19.633465503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:14.793320099 +0000 UTC m=+22.715587869" watchObservedRunningTime="2026-07-31 22:57:14.810782872 +0000 UTC m=+22.733050639" Jul 31 22:57:14.811292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.811271 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xzgjv" podStartSLOduration=2.294085149 podStartE2EDuration="21.811265713s" totalImagesPullingTime="19.517180564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:14.810917674 +0000 UTC m=+22.733185426" watchObservedRunningTime="2026-07-31 22:57:14.811265713 +0000 UTC m=+22.733533485" Jul 31 22:57:14.864437 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:14.864393 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vmspv" podStartSLOduration=2.226186198 podStartE2EDuration="21.864376757s" totalImagesPullingTime="19.638190559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:14.864095784 +0000 UTC m=+22.786363555" watchObservedRunningTime="2026-07-31 22:57:14.864376757 +0000 UTC m=+22.786644528" Jul 31 22:57:15.532875 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.532853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:15.533032 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.532853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:15.533032 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.532853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:15.533100 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:15.532934 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:15.533136 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:15.533019 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:15.533136 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:15.533098 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:15.654259 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.654104 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 22:57:15.784924 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.784841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jdfm" event={"ID":"832f6392-a691-4432-a15b-d120dfc94edc","Type":"ContainerStarted","Data":"20c9480a9c0a6a5a6ccf460c2dfb25a8a6ef186ca2d7e0575320af2b185befb8"} Jul 31 22:57:15.786592 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.786563 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" event={"ID":"c67ad40e-3221-4384-bbe2-a000c3b61847","Type":"ContainerStarted","Data":"8aa143455a01a556315acc1365ecda2ef58a135d20345266927938a86e79b159"} Jul 31 22:57:15.821246 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.821202 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7jdfm" podStartSLOduration=22.821186468 podStartE2EDuration="22.821186468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-31 22:57:15.80675535 +0000 UTC m=+23.729023119" watchObservedRunningTime="2026-07-31 22:57:15.821186468 +0000 UTC m=+23.743454237" Jul 31 22:57:15.950756 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.950734 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:57:15.951255 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:15.951213 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:57:16.486025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.485750 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T22:57:15.654255447Z","UUID":"5a4b1b57-076f-45b9-bf5d-d83a38fe9b20","Handler":null,"Name":"","Endpoint":""} Jul 31 22:57:16.487446 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.487424 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 22:57:16.487578 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.487466 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 22:57:16.790775 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.790748 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" event={"ID":"6acb4bf471420828b96cf11e2a7009d4","Type":"ContainerStarted","Data":"85ee50254f6bd2f13eb17592eb4cc1ebd97910a2fe7668aefb827bbd7499dd42"} Jul 31 22:57:16.793936 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.793916 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 22:57:16.794483 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.794459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"944e13db818edd6ccbd8341bb9b421c03bbc3130406088e48726beb87a937198"} Jul 31 22:57:16.806877 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:16.806832 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-58.ec2.internal" podStartSLOduration=22.806799822 podStartE2EDuration="22.806799822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:16.806084065 +0000 UTC m=+24.728351836" watchObservedRunningTime="2026-07-31 22:57:16.806799822 +0000 UTC m=+24.729067596" Jul 31 22:57:17.532870 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.532843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:17.533045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.532843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:17.533045 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:17.532941 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:17.533045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.532974 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:17.535619 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:17.533497 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:17.535619 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:17.533693 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:17.798452 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.798387 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 31 22:57:17.798452 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.798380 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" event={"ID":"c67ad40e-3221-4384-bbe2-a000c3b61847","Type":"ContainerStarted","Data":"b04ff72f1f185873d08f00509ebbfa10e7fa3c1de35ee0eaa0d155230d41a520"} Jul 31 22:57:17.820359 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.820315 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qlxtq" podStartSLOduration=2.9552374219999997 podStartE2EDuration="24.820303906s" totalImagesPullingTime="21.865066484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:17.818814999 +0000 UTC m=+25.741082768" watchObservedRunningTime="2026-07-31 22:57:17.820303906 +0000 UTC m=+25.742571672" Jul 31 22:57:17.835632 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.835614 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:57:17.836163 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:17.836144 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vmspv" Jul 31 22:57:18.803700 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:18.803540 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 22:57:18.804142 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:18.804062 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"06883adc0e5d7c4e35a8117f8b4864700c676f9de8fa65611f49ce2da2874bfc"} Jul 31 22:57:18.804644 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:18.804626 2570 scope.go:117] "RemoveContainer" containerID="8514eb2bd7b4cdf1fba5945c47b4341c8d3cfb8a3bed1303586760c48bf4c09d" Jul 31 22:57:19.532865 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.532837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:19.533033 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.532840 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:19.533033 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:19.532969 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:19.533033 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.532848 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:19.535509 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:19.533043 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:19.535509 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:19.533617 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:19.813435 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.813354 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 22:57:19.817590 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.815602 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" event={"ID":"23ac3832-68eb-46ab-910a-d87d76403e76","Type":"ContainerStarted","Data":"a9a34bc8f93f8e8b4066bd5c3fa765675de2c72a5b14291c7deeea965b1d9db9"} Jul 31 22:57:19.817590 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.815803 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:19.817590 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.815972 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:19.817590 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.816039 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:19.836413 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.836390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:19.837659 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.837639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:19.853359 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:19.853317 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" podStartSLOduration=7.241537698 podStartE2EDuration="26.853305249s" totalImagesPullingTime="19.611767551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:19.853043588 +0000 UTC m=+27.775311370" watchObservedRunningTime="2026-07-31 22:57:19.853305249 +0000 UTC m=+27.775573018" Jul 31 22:57:20.496868 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.496841 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9ftrs"] Jul 31 22:57:20.497037 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.496968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:20.497095 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:20.497069 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:20.498416 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.498395 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rrsk"] Jul 31 22:57:20.498507 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.498490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:20.498606 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:20.498586 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:20.501741 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.501719 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvnqs"] Jul 31 22:57:20.501834 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:20.501809 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:20.501920 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:20.501897 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:21.532426 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:21.532399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:21.533008 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:21.532503 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:21.820577 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:21.820505 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="938fec3335181ca16694837a564b9fdd0d93355c42cb02077be90b5b707be024" exitCode=0 Jul 31 22:57:21.820701 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:21.820583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"938fec3335181ca16694837a564b9fdd0d93355c42cb02077be90b5b707be024"} Jul 31 22:57:22.533699 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:22.533531 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:22.534100 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:22.533573 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:22.534100 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:22.533791 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:22.534100 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:22.533908 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:22.824738 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:22.824712 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="d21fc570080e4d2d15cf7566fe3c163a6dc0e71436b05be0a3653030edf4b32d" exitCode=0 Jul 31 22:57:22.824880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:22.824758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"d21fc570080e4d2d15cf7566fe3c163a6dc0e71436b05be0a3653030edf4b32d"} Jul 31 22:57:23.532836 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:23.532791 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:23.533002 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:23.532916 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:23.829105 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:23.829045 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="5b47a02b59155c8e62c4199b816e2f84f0c984f9d29f88b23076c475429162e9" exitCode=0 Jul 31 22:57:23.829105 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:23.829098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"5b47a02b59155c8e62c4199b816e2f84f0c984f9d29f88b23076c475429162e9"} Jul 31 22:57:24.532857 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:24.532810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:24.533029 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:24.532814 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:24.533029 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:24.532941 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvnqs" podUID="3fb26301-9bdf-4865-a8fa-1e1a00d4246c" Jul 31 22:57:24.533156 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:24.533026 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rrsk" podUID="091a5632-061d-4b31-8e30-f03fffa6a68d" Jul 31 22:57:25.316607 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:25.316574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:25.317028 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:25.316629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:25.317028 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.316717 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:25.317028 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.316777 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret podName:091a5632-061d-4b31-8e30-f03fffa6a68d nodeName:}" failed. No retries permitted until 2026-07-31 22:57:57.316763101 +0000 UTC m=+65.239030852 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret") pod "global-pull-secret-syncer-2rrsk" (UID: "091a5632-061d-4b31-8e30-f03fffa6a68d") : object "kube-system"/"original-pull-secret" not registered Jul 31 22:57:25.317028 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.316719 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:25.317028 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.316881 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:57:57.316863684 +0000 UTC m=+65.239131449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 22:57:25.417649 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:25.417621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:25.417785 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.417765 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 22:57:25.417785 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.417785 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 22:57:25.417871 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.417794 2570 projected.go:196] Error preparing data for projected volume kube-api-access-jqmlp for pod openshift-network-diagnostics/network-check-target-qvnqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:25.417871 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.417850 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp podName:3fb26301-9bdf-4865-a8fa-1e1a00d4246c nodeName:}" failed. No retries permitted until 2026-07-31 22:57:57.417838266 +0000 UTC m=+65.340106014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqmlp" (UniqueName: "kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp") pod "network-check-target-qvnqs" (UID: "3fb26301-9bdf-4865-a8fa-1e1a00d4246c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 22:57:25.532954 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:25.532926 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:25.533110 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:25.533044 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ftrs" podUID="842f999d-a623-4c09-8218-099a42fe84cc" Jul 31 22:57:26.446559 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.446533 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-58.ec2.internal" event="NodeReady" Jul 31 22:57:26.446963 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.446657 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 22:57:26.497195 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.497162 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:57:26.534974 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.534950 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87fdb"] Jul 31 22:57:26.535094 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.535069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.538721 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.538698 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 31 22:57:26.538855 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.538744 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 31 22:57:26.538855 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.538748 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 31 22:57:26.538990 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.538960 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sn4bg\"" Jul 31 22:57:26.545394 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.545372 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 31 22:57:26.546464 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.546444 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pg6g5"] Jul 31 22:57:26.546612 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.546529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.546612 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.546555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:26.546809 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.546776 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:26.549422 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549226 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 22:57:26.549422 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549267 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 22:57:26.549623 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549534 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 22:57:26.549623 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549604 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 22:57:26.549805 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549639 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 22:57:26.550197 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.549956 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c9ph2\"" Jul 31 22:57:26.550256 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.550214 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtx5g\"" Jul 31 22:57:26.550433 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.550400 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 22:57:26.560526 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.560508 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:57:26.560609 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.560540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.564218 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.564202 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7jxx7\"" Jul 31 22:57:26.564637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.564620 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87fdb"] Jul 31 22:57:26.564708 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.564659 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pg6g5"] Jul 31 22:57:26.565308 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.564783 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 22:57:26.565308 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.564808 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 22:57:26.625991 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.625969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxvk\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.625992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626237 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626237 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.626237 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.626166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727136 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e665f6f-530c-4b2b-967b-8d4be419d54a-config-volume\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.727136 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727285 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727285 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e665f6f-530c-4b2b-967b-8d4be419d54a-tmp-dir\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.727285 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:26.727228 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:26.727285 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:26.727242 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:26.727458 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:26.727299 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:27.227278122 +0000 UTC m=+35.149545873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:26.727458 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727458 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f9762d-6540-44e2-bddc-db63f798f2b5-cert\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.727458 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727655 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.727655 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/50f9762d-6540-44e2-bddc-db63f798f2b5-kube-api-access-j6chz\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.727655 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727655 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727655 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.727928 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqz8\" (UniqueName: \"kubernetes.io/projected/1e665f6f-530c-4b2b-967b-8d4be419d54a-kube-api-access-vsqz8\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.727928 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.727749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxvk\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.728045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.728023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.728045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.728034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.728668 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.728648 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.732284 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.732175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.732389 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.732201 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.738589 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.738564 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.738935 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.738915 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxvk\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:26.828482 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e665f6f-530c-4b2b-967b-8d4be419d54a-config-volume\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.828614 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e665f6f-530c-4b2b-967b-8d4be419d54a-tmp-dir\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.828614 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f9762d-6540-44e2-bddc-db63f798f2b5-cert\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.828614 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.828614 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/50f9762d-6540-44e2-bddc-db63f798f2b5-kube-api-access-j6chz\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.828745 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqz8\" (UniqueName: \"kubernetes.io/projected/1e665f6f-530c-4b2b-967b-8d4be419d54a-kube-api-access-vsqz8\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.828952 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.828935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e665f6f-530c-4b2b-967b-8d4be419d54a-config-volume\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.829185 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.829163 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e665f6f-530c-4b2b-967b-8d4be419d54a-tmp-dir\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.829263 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:26.829247 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:26.829312 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:26.829299 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:27.329283373 +0000 UTC m=+35.251551126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:26.831515 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.831496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f9762d-6540-44e2-bddc-db63f798f2b5-cert\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.838043 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.838024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqz8\" (UniqueName: \"kubernetes.io/projected/1e665f6f-530c-4b2b-967b-8d4be419d54a-kube-api-access-vsqz8\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:26.838120 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.838054 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6chz\" (UniqueName: \"kubernetes.io/projected/50f9762d-6540-44e2-bddc-db63f798f2b5-kube-api-access-j6chz\") pod \"ingress-canary-87fdb\" (UID: \"50f9762d-6540-44e2-bddc-db63f798f2b5\") " pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:26.858108 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:26.858084 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87fdb" Jul 31 22:57:27.028137 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.028107 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87fdb"] Jul 31 22:57:27.232250 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.232175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:27.232472 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:27.232340 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:27.232472 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:27.232360 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:27.232472 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:27.232423 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:28.23240496 +0000 UTC m=+36.154672727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:27.333199 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.333165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:27.333430 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:27.333333 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:27.333430 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:27.333395 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:28.333381475 +0000 UTC m=+36.255649223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:27.533334 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.533264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:27.536152 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.536112 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 22:57:27.536280 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:27.536211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8t45\"" Jul 31 22:57:28.240425 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:28.240381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:28.240613 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:28.240563 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:28.240613 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:28.240588 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:28.240713 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:28.240660 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:30.240639058 +0000 UTC m=+38.162906807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:28.341223 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:28.341188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:28.341385 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:28.341319 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:28.341385 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:28.341375 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:30.341359003 +0000 UTC m=+38.263626778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:28.472574 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:28.472534 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f9762d_6540_44e2_bddc_db63f798f2b5.slice/crio-ccbf2ff7876868461430e8811025efad4f24179995de3e37b09988488e9689dd WatchSource:0}: Error finding container ccbf2ff7876868461430e8811025efad4f24179995de3e37b09988488e9689dd: Status 404 returned error can't find the container with id ccbf2ff7876868461430e8811025efad4f24179995de3e37b09988488e9689dd Jul 31 22:57:28.842163 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:28.842125 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87fdb" event={"ID":"50f9762d-6540-44e2-bddc-db63f798f2b5","Type":"ContainerStarted","Data":"ccbf2ff7876868461430e8811025efad4f24179995de3e37b09988488e9689dd"} Jul 31 22:57:29.846694 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:29.846661 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="aeb95da0c9e112c1c8023cfd26a0003f2710f6c6e968cded5c71286fcee7fc25" exitCode=0 Jul 31 22:57:29.847180 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:29.846707 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"aeb95da0c9e112c1c8023cfd26a0003f2710f6c6e968cded5c71286fcee7fc25"} Jul 31 22:57:30.255583 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:30.255552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:30.255742 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:30.255675 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:30.255742 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:30.255687 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:30.255742 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:30.255735 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:34.255721071 +0000 UTC m=+42.177988819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:30.356457 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:30.356431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:30.356591 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:30.356574 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:30.356643 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:30.356637 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:34.356620934 +0000 UTC m=+42.278888682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:30.851325 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:30.851287 2570 generic.go:356] "Generic (PLEG): container finished" podID="1871d052-b938-4d33-9aa5-cdad64a19843" containerID="cbee0816a0c3169eeeef09f02c88c9ca65f63dc757dd6882690bef8bf1e6c503" exitCode=0 Jul 31 22:57:30.851760 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:30.851332 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerDied","Data":"cbee0816a0c3169eeeef09f02c88c9ca65f63dc757dd6882690bef8bf1e6c503"} Jul 31 22:57:31.856692 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:31.856656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" event={"ID":"1871d052-b938-4d33-9aa5-cdad64a19843","Type":"ContainerStarted","Data":"03ebbf6380dd3f912089351a391b0741ffb80c4b441e05610aab04f8b6b7f55e"} Jul 31 22:57:31.858075 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:31.858052 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87fdb" event={"ID":"50f9762d-6540-44e2-bddc-db63f798f2b5","Type":"ContainerStarted","Data":"cf62c77ce57905db61bb433241923010e282deeb853bb88d54dace6b1175e49e"} Jul 31 22:57:31.883776 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:31.883732 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2rcc8" podStartSLOduration=6.657612389 podStartE2EDuration="38.883721377s" totalImagesPullingTime="32.226108988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:31.883574176 +0000 UTC m=+39.805841947" watchObservedRunningTime="2026-07-31 22:57:31.883721377 +0000 UTC m=+39.805989146" Jul 31 22:57:31.902009 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:31.901979 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87fdb" podStartSLOduration=3.415953039 podStartE2EDuration="5.901968874s" totalImagesPullingTime="2.486015835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:31.900921464 +0000 UTC m=+39.823189233" watchObservedRunningTime="2026-07-31 22:57:31.901968874 +0000 UTC m=+39.824236642" Jul 31 22:57:34.282410 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:34.282368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:34.282793 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:34.282525 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:34.282793 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:34.282547 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:34.282793 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:34.282602 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:42.282587132 +0000 UTC m=+50.204854884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:34.383234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:34.383207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:34.383355 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:34.383301 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:34.383355 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:34.383347 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:42.383335172 +0000 UTC m=+50.305602920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:42.336422 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.336384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:42.336766 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:42.336530 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 22:57:42.336766 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:42.336548 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b56b88d89-tdg28: secret "image-registry-tls" not found Jul 31 22:57:42.336766 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:42.336602 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls podName:6fb1b3ae-60b2-4cb9-9499-38b7e750c488 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:58.336587615 +0000 UTC m=+66.258855363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls") pod "image-registry-7b56b88d89-tdg28" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488") : secret "image-registry-tls" not found Jul 31 22:57:42.436880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.436857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:42.436977 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:42.436957 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 22:57:42.437012 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:42.437005 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls podName:1e665f6f-530c-4b2b-967b-8d4be419d54a nodeName:}" failed. No retries permitted until 2026-07-31 22:57:58.436989665 +0000 UTC m=+66.359257416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls") pod "dns-default-pg6g5" (UID: "1e665f6f-530c-4b2b-967b-8d4be419d54a") : secret "dns-default-metrics-tls" not found Jul 31 22:57:42.762500 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.762468 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42"] Jul 31 22:57:42.795952 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.795929 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hwdkz"] Jul 31 22:57:42.796093 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.796051 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" Jul 31 22:57:42.798538 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.798518 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 31 22:57:42.798801 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.798788 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 31 22:57:42.799673 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.799659 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5fcwt\"" Jul 31 22:57:42.819642 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.819622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.819642 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.819624 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42"] Jul 31 22:57:42.819753 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.819673 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hwdkz"] Jul 31 22:57:42.822021 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.822005 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rgl8d\"" Jul 31 22:57:42.822102 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.822006 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 31 22:57:42.822546 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.822525 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 31 22:57:42.823403 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.823387 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 31 22:57:42.823921 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.823903 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 31 22:57:42.827457 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.827437 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 31 22:57:42.846902 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.846881 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8"] Jul 31 22:57:42.857421 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.857400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78954292-2379-439c-babe-b86b2156e4a1-serving-cert\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.857495 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.857427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-config\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.857495 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.857447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4lw\" (UniqueName: \"kubernetes.io/projected/4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee-kube-api-access-wf4lw\") pod \"volume-data-source-validator-5b8776669-wtf42\" (UID: \"4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" Jul 31 22:57:42.857495 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.857478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxlz\" (UniqueName: \"kubernetes.io/projected/78954292-2379-439c-babe-b86b2156e4a1-kube-api-access-8dxlz\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.857663 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.857605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-trusted-ca\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.867087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.867071 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8"] Jul 31 22:57:42.867179 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.867156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" Jul 31 22:57:42.870159 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.870143 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c7stz\"" Jul 31 22:57:42.958543 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxlz\" (UniqueName: \"kubernetes.io/projected/78954292-2379-439c-babe-b86b2156e4a1-kube-api-access-8dxlz\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.958626 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-trusted-ca\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.958626 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609-kube-api-access-8g2dv\") pod \"network-check-source-65bdbbff45-7r4b8\" (UID: \"6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" Jul 31 22:57:42.958705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78954292-2379-439c-babe-b86b2156e4a1-serving-cert\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.958705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-config\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.958705 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.958697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4lw\" (UniqueName: \"kubernetes.io/projected/4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee-kube-api-access-wf4lw\") pod \"volume-data-source-validator-5b8776669-wtf42\" (UID: \"4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" Jul 31 22:57:42.959374 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.959356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-config\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.959478 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.959458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78954292-2379-439c-babe-b86b2156e4a1-trusted-ca\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.962026 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.962006 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78954292-2379-439c-babe-b86b2156e4a1-serving-cert\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:42.973320 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.973292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4lw\" (UniqueName: \"kubernetes.io/projected/4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee-kube-api-access-wf4lw\") pod \"volume-data-source-validator-5b8776669-wtf42\" (UID: \"4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" Jul 31 22:57:42.973538 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:42.973518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxlz\" (UniqueName: \"kubernetes.io/projected/78954292-2379-439c-babe-b86b2156e4a1-kube-api-access-8dxlz\") pod \"console-operator-5ccd6b445f-hwdkz\" (UID: \"78954292-2379-439c-babe-b86b2156e4a1\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:43.059740 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.059691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609-kube-api-access-8g2dv\") pod \"network-check-source-65bdbbff45-7r4b8\" (UID: \"6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" Jul 31 22:57:43.068252 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.068229 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609-kube-api-access-8g2dv\") pod \"network-check-source-65bdbbff45-7r4b8\" (UID: \"6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" Jul 31 22:57:43.104399 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.104383 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" Jul 31 22:57:43.127888 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.127861 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:43.174927 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.174876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" Jul 31 22:57:43.237643 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.237476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42"] Jul 31 22:57:43.273193 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.273165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hwdkz"] Jul 31 22:57:43.276293 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:43.276273 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78954292_2379_439c_babe_b86b2156e4a1.slice/crio-48b574a05b83bd3787a5421a8e57e4f01d9515164ebc0486d9c5c02cbd695259 WatchSource:0}: Error finding container 48b574a05b83bd3787a5421a8e57e4f01d9515164ebc0486d9c5c02cbd695259: Status 404 returned error can't find the container with id 48b574a05b83bd3787a5421a8e57e4f01d9515164ebc0486d9c5c02cbd695259 Jul 31 22:57:43.311882 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.311803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8"] Jul 31 22:57:43.314961 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:43.314939 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3e3e29_d1c3_4ff0_8d0c_1c2e5e8ef609.slice/crio-b3ab35bf3dc79ad65c8b6bf4469c52c3501f58044b2c3e6f7e36d7acb8e50be4 WatchSource:0}: Error finding container b3ab35bf3dc79ad65c8b6bf4469c52c3501f58044b2c3e6f7e36d7acb8e50be4: Status 404 returned error can't find the container with id b3ab35bf3dc79ad65c8b6bf4469c52c3501f58044b2c3e6f7e36d7acb8e50be4 Jul 31 22:57:43.885166 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.885131 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" event={"ID":"6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609","Type":"ContainerStarted","Data":"b3ab35bf3dc79ad65c8b6bf4469c52c3501f58044b2c3e6f7e36d7acb8e50be4"} Jul 31 22:57:43.886506 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.886480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerStarted","Data":"48b574a05b83bd3787a5421a8e57e4f01d9515164ebc0486d9c5c02cbd695259"} Jul 31 22:57:43.887659 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:43.887629 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" event={"ID":"4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee","Type":"ContainerStarted","Data":"4536f5f58811f7e2d4390e4918a633e9ba2f2061d6e4668685caf1c5768648e8"} Jul 31 22:57:44.891932 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:44.891897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" event={"ID":"4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee","Type":"ContainerStarted","Data":"5adc8f6d1d27641eea45c74fcbb334984c7c59edbfa43ab2fdf34caa35fda8a3"} Jul 31 22:57:44.910931 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:44.910885 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-wtf42" podStartSLOduration=1.363936861 podStartE2EDuration="2.910871618s" totalImagesPullingTime="1.546934757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:44.910441088 +0000 UTC m=+52.832708861" watchObservedRunningTime="2026-07-31 22:57:44.910871618 +0000 UTC m=+52.833139415" Jul 31 22:57:45.895690 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:45.895625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/0.log" Jul 31 22:57:45.895690 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:45.895665 2570 generic.go:356] "Generic (PLEG): container finished" podID="78954292-2379-439c-babe-b86b2156e4a1" containerID="10906dc3248fb719c6397f79cdcce55158e13330f98a5d4065b9202e74fbcee2" exitCode=255 Jul 31 22:57:45.896132 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:45.895698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerDied","Data":"10906dc3248fb719c6397f79cdcce55158e13330f98a5d4065b9202e74fbcee2"} Jul 31 22:57:45.896132 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:45.896032 2570 scope.go:117] "RemoveContainer" containerID="10906dc3248fb719c6397f79cdcce55158e13330f98a5d4065b9202e74fbcee2" Jul 31 22:57:46.901874 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.901783 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" event={"ID":"6d3e3e29-d1c3-4ff0-8d0c-1c2e5e8ef609","Type":"ContainerStarted","Data":"2096213d22e9fd2483e338b6da506c09f896bab284d4eea880f80a815a72d16c"} Jul 31 22:57:46.903800 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.903780 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/0.log" Jul 31 22:57:46.903924 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.903872 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerStarted","Data":"b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be"} Jul 31 22:57:46.904063 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.904049 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:46.905527 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.905503 2570 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-hwdkz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Jul 31 22:57:46.905652 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.905546 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Jul 31 22:57:46.917788 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.917742 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7r4b8" podStartSLOduration=1.4113166129999999 podStartE2EDuration="4.917728931s" totalImagesPullingTime="3.506412318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:46.917044772 +0000 UTC m=+54.839312542" watchObservedRunningTime="2026-07-31 22:57:46.917728931 +0000 UTC m=+54.839996679" Jul 31 22:57:46.934492 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:46.934459 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podStartSLOduration=2.765903791 podStartE2EDuration="4.934448543s" totalImagesPullingTime="2.168544752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:46.93409706 +0000 UTC m=+54.856364831" watchObservedRunningTime="2026-07-31 22:57:46.934448543 +0000 UTC m=+54.856716313" Jul 31 22:57:47.915040 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915012 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/1.log" Jul 31 22:57:47.915492 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915474 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/0.log" Jul 31 22:57:47.915538 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915512 2570 generic.go:356] "Generic (PLEG): container finished" podID="78954292-2379-439c-babe-b86b2156e4a1" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" exitCode=255 Jul 31 22:57:47.915629 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerDied","Data":"b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be"} Jul 31 22:57:47.915707 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915695 2570 scope.go:117] "RemoveContainer" containerID="10906dc3248fb719c6397f79cdcce55158e13330f98a5d4065b9202e74fbcee2" Jul 31 22:57:47.915975 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:47.915950 2570 scope.go:117] "RemoveContainer" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" Jul 31 22:57:47.916134 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:47.916115 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:57:48.537986 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.537932 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mw7k9"] Jul 31 22:57:48.540439 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.540421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.543124 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.543097 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 22:57:48.543260 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.543144 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 22:57:48.544259 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.544241 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 22:57:48.544259 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.544255 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmjpj\"" Jul 31 22:57:48.544368 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.544276 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 22:57:48.555916 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.555894 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mw7k9"] Jul 31 22:57:48.604456 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.604425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-crio-socket\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.604550 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.604460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-data-volume\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.604604 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.604583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.604643 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.604605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.604739 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.604645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9wx\" (UniqueName: \"kubernetes.io/projected/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-api-access-2q9wx\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.705735 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705699 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.705849 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.705849 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9wx\" (UniqueName: \"kubernetes.io/projected/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-api-access-2q9wx\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.705849 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:48.705805 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705858 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-crio-socket\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:48.705886 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:49.205871837 +0000 UTC m=+57.128139624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : configmap "kube-rbac-proxy" not found Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-data-volume\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.705930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-crio-socket\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:48.705905 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:57:48.706027 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:48.706027 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:49.206011444 +0000 UTC m=+57.128279206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:57:48.706211 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.706195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-data-volume\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.716257 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.716239 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9wx\" (UniqueName: \"kubernetes.io/projected/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-api-access-2q9wx\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:48.919625 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.919568 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/1.log" Jul 31 22:57:48.919946 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:48.919845 2570 scope.go:117] "RemoveContainer" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" Jul 31 22:57:48.920000 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:48.919982 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:57:49.166756 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.166721 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22"] Jul 31 22:57:49.168981 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.168964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" Jul 31 22:57:49.171596 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.171529 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tkh8b\"" Jul 31 22:57:49.171942 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.171925 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 31 22:57:49.172002 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.171955 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 31 22:57:49.178651 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.178629 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22"] Jul 31 22:57:49.210391 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.210365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:49.210391 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.210392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:49.210536 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.210412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkcft\" (UniqueName: \"kubernetes.io/projected/fe6aaca1-cd56-4bb5-ac30-cabb3027fa58-kube-api-access-fkcft\") pod \"migrator-68bc745d87-hfz22\" (UID: \"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" Jul 31 22:57:49.210536 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:49.210482 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 22:57:49.210536 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:49.210507 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:57:49.210642 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:49.210546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:50.210533373 +0000 UTC m=+58.132801125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : configmap "kube-rbac-proxy" not found Jul 31 22:57:49.210642 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:49.210559 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:50.210553514 +0000 UTC m=+58.132821262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:57:49.310748 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.310725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkcft\" (UniqueName: \"kubernetes.io/projected/fe6aaca1-cd56-4bb5-ac30-cabb3027fa58-kube-api-access-fkcft\") pod \"migrator-68bc745d87-hfz22\" (UID: \"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" Jul 31 22:57:49.322989 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.322965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkcft\" (UniqueName: \"kubernetes.io/projected/fe6aaca1-cd56-4bb5-ac30-cabb3027fa58-kube-api-access-fkcft\") pod \"migrator-68bc745d87-hfz22\" (UID: \"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" Jul 31 22:57:49.477107 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.477084 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" Jul 31 22:57:49.593417 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.593386 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22"] Jul 31 22:57:49.596113 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:49.596089 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6aaca1_cd56_4bb5_ac30_cabb3027fa58.slice/crio-c89a7efd9e5473e2c8e83096988df2fde4a10637451b6566cdf9cc45d3b88f54 WatchSource:0}: Error finding container c89a7efd9e5473e2c8e83096988df2fde4a10637451b6566cdf9cc45d3b88f54: Status 404 returned error can't find the container with id c89a7efd9e5473e2c8e83096988df2fde4a10637451b6566cdf9cc45d3b88f54 Jul 31 22:57:49.922643 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:49.922568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" event={"ID":"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58","Type":"ContainerStarted","Data":"c89a7efd9e5473e2c8e83096988df2fde4a10637451b6566cdf9cc45d3b88f54"} Jul 31 22:57:50.216860 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.216829 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:50.216997 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.216872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:50.216997 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:50.216952 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 22:57:50.217080 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:50.217003 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:57:50.217080 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:50.217026 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:52.217008273 +0000 UTC m=+60.139276033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : configmap "kube-rbac-proxy" not found Jul 31 22:57:50.217080 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:50.217042 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:52.217034464 +0000 UTC m=+60.139302212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:57:50.284861 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.284840 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzgjv_9fcb9bc0-23e8-4f43-9468-54fb1c39bd23/dns-node-resolver/0.log" Jul 31 22:57:50.885080 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.885052 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rfbw_7dd0530a-f47c-4900-95e6-e9fec3c7924d/node-ca/0.log" Jul 31 22:57:50.985299 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.985271 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bzw2x"] Jul 31 22:57:50.999979 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:50.999951 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bzw2x"] Jul 31 22:57:51.000108 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.000070 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.002714 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.002686 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ktz49\"" Jul 31 22:57:51.002969 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.002953 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 31 22:57:51.003800 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.003787 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 31 22:57:51.003986 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.003970 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 31 22:57:51.007840 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.004395 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 31 22:57:51.022774 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.022756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-cabundle\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.022892 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.022847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpbq\" (UniqueName: \"kubernetes.io/projected/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-kube-api-access-zgpbq\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.022892 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.022879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-key\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.124096 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.124076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpbq\" (UniqueName: \"kubernetes.io/projected/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-kube-api-access-zgpbq\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.124168 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.124130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-key\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.124199 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.124171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-cabundle\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.124716 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.124695 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-cabundle\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.126356 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.126336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-signing-key\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.132776 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.132757 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpbq\" (UniqueName: \"kubernetes.io/projected/6692ce0e-1dae-4b1b-82c8-ec4842fcede9-kube-api-access-zgpbq\") pod \"service-ca-98c88fbb5-bzw2x\" (UID: \"6692ce0e-1dae-4b1b-82c8-ec4842fcede9\") " pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.311356 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.311328 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" Jul 31 22:57:51.438044 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.438015 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-bzw2x"] Jul 31 22:57:51.441372 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:51.441346 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6692ce0e_1dae_4b1b_82c8_ec4842fcede9.slice/crio-251b96b4f22e5808b32369b2c64c74cde31264721292cd2597bd5939d74e8207 WatchSource:0}: Error finding container 251b96b4f22e5808b32369b2c64c74cde31264721292cd2597bd5939d74e8207: Status 404 returned error can't find the container with id 251b96b4f22e5808b32369b2c64c74cde31264721292cd2597bd5939d74e8207 Jul 31 22:57:51.686174 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.686123 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87fdb_50f9762d-6540-44e2-bddc-db63f798f2b5/serve-healthcheck-canary/0.log" Jul 31 22:57:51.836764 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.836744 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nv77q" Jul 31 22:57:51.929530 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.929503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" event={"ID":"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58","Type":"ContainerStarted","Data":"a3ba0b036b3785f77a22c29f0d79100b46f459a28076038a3fd09db7b77dfab6"} Jul 31 22:57:51.929649 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.929553 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" event={"ID":"fe6aaca1-cd56-4bb5-ac30-cabb3027fa58","Type":"ContainerStarted","Data":"e450068fca13e8c2627e7211c5a635e2c3bee53cf6c850130c3c9c2c9f5e9f0f"} Jul 31 22:57:51.931880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.931857 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" event={"ID":"6692ce0e-1dae-4b1b-82c8-ec4842fcede9","Type":"ContainerStarted","Data":"251b96b4f22e5808b32369b2c64c74cde31264721292cd2597bd5939d74e8207"} Jul 31 22:57:51.947785 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:51.947741 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-hfz22" podStartSLOduration=1.426136866 podStartE2EDuration="2.94772702s" totalImagesPullingTime="1.521590154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:51.946705667 +0000 UTC m=+59.868973437" watchObservedRunningTime="2026-07-31 22:57:51.94772702 +0000 UTC m=+59.869994788" Jul 31 22:57:52.231857 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:52.231767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:52.232271 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:52.231883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:52.232271 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:52.231943 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:57:52.232271 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:52.231991 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 22:57:52.232271 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:52.232013 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:56.231993245 +0000 UTC m=+64.154260993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:57:52.232271 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:52.232047 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:57:56.232035278 +0000 UTC m=+64.154303026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : configmap "kube-rbac-proxy" not found Jul 31 22:57:53.128599 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:53.128554 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:57:53.129050 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:53.129032 2570 scope.go:117] "RemoveContainer" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" Jul 31 22:57:53.129239 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:53.129215 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:57:53.939087 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:53.939048 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" event={"ID":"6692ce0e-1dae-4b1b-82c8-ec4842fcede9","Type":"ContainerStarted","Data":"f6c4ddf82591668ebb2eb77fb74881cccc553c9644e01d85af05d32b7bff2758"} Jul 31 22:57:53.972787 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:53.972736 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-bzw2x" podStartSLOduration=2.181535611 podStartE2EDuration="3.972718909s" totalImagesPullingTime="1.791183298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:53.972328627 +0000 UTC m=+61.894596398" watchObservedRunningTime="2026-07-31 22:57:53.972718909 +0000 UTC m=+61.894986681" Jul 31 22:57:56.263687 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:56.263650 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:56.264219 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:56.263733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:57:56.264219 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:56.263789 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 22:57:56.264219 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:56.263874 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:58:04.263859751 +0000 UTC m=+72.186127503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : configmap "kube-rbac-proxy" not found Jul 31 22:57:56.264219 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:56.263937 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:57:56.264219 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:56.264060 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:58:04.264040623 +0000 UTC m=+72.186308384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:57:57.372861 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.372806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:57.373292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.372946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:57:57.375401 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.375382 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 22:57:57.375595 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.375582 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 22:57:57.383758 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:57.383742 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 22:57:57.383829 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:57:57.383796 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs podName:842f999d-a623-4c09-8218-099a42fe84cc nodeName:}" failed. No retries permitted until 2026-07-31 22:59:01.383778149 +0000 UTC m=+129.306045900 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs") pod "network-metrics-daemon-9ftrs" (UID: "842f999d-a623-4c09-8218-099a42fe84cc") : secret "metrics-daemon-secret" not found Jul 31 22:57:57.385750 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.385727 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/091a5632-061d-4b31-8e30-f03fffa6a68d-original-pull-secret\") pod \"global-pull-secret-syncer-2rrsk\" (UID: \"091a5632-061d-4b31-8e30-f03fffa6a68d\") " pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:57.473986 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.473962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:57.476196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.476179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmlp\" (UniqueName: \"kubernetes.io/projected/3fb26301-9bdf-4865-a8fa-1e1a00d4246c-kube-api-access-jqmlp\") pod \"network-check-target-qvnqs\" (UID: \"3fb26301-9bdf-4865-a8fa-1e1a00d4246c\") " pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:57.476905 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.476880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rrsk" Jul 31 22:57:57.599634 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.599604 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rrsk"] Jul 31 22:57:57.602282 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:57.602253 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091a5632_061d_4b31_8e30_f03fffa6a68d.slice/crio-9de3982d719f6d3f81751369216df1c3139cfa228667f7957599c446dce44dff WatchSource:0}: Error finding container 9de3982d719f6d3f81751369216df1c3139cfa228667f7957599c446dce44dff: Status 404 returned error can't find the container with id 9de3982d719f6d3f81751369216df1c3139cfa228667f7957599c446dce44dff Jul 31 22:57:57.772113 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.772093 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtx5g\"" Jul 31 22:57:57.780469 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.780447 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:57.925573 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.925543 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvnqs"] Jul 31 22:57:57.928283 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:57.928255 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb26301_9bdf_4865_a8fa_1e1a00d4246c.slice/crio-571fb0762f892b2be6670e81192fa8c021e0ac39f46d2f8336e2f79b7ef51e80 WatchSource:0}: Error finding container 571fb0762f892b2be6670e81192fa8c021e0ac39f46d2f8336e2f79b7ef51e80: Status 404 returned error can't find the container with id 571fb0762f892b2be6670e81192fa8c021e0ac39f46d2f8336e2f79b7ef51e80 Jul 31 22:57:57.952521 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.952486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvnqs" event={"ID":"3fb26301-9bdf-4865-a8fa-1e1a00d4246c","Type":"ContainerStarted","Data":"571fb0762f892b2be6670e81192fa8c021e0ac39f46d2f8336e2f79b7ef51e80"} Jul 31 22:57:57.953544 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:57.953525 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rrsk" event={"ID":"091a5632-061d-4b31-8e30-f03fffa6a68d","Type":"ContainerStarted","Data":"9de3982d719f6d3f81751369216df1c3139cfa228667f7957599c446dce44dff"} Jul 31 22:57:58.381968 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.381900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:58.384368 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.384344 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"image-registry-7b56b88d89-tdg28\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:58.482861 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.482749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:58.485122 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.485099 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e665f6f-530c-4b2b-967b-8d4be419d54a-metrics-tls\") pod \"dns-default-pg6g5\" (UID: \"1e665f6f-530c-4b2b-967b-8d4be419d54a\") " pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:58.648078 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.648012 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sn4bg\"" Jul 31 22:57:58.655767 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.655736 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:58.686068 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.686050 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7jxx7\"" Jul 31 22:57:58.693484 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.693457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pg6g5" Jul 31 22:57:58.838449 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.838397 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:57:58.841105 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:58.841073 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb1b3ae_60b2_4cb9_9499_38b7e750c488.slice/crio-bb06e003963fb93b1c900e3dfd1606691a827b43d5f969f5232b0fde974a7c45 WatchSource:0}: Error finding container bb06e003963fb93b1c900e3dfd1606691a827b43d5f969f5232b0fde974a7c45: Status 404 returned error can't find the container with id bb06e003963fb93b1c900e3dfd1606691a827b43d5f969f5232b0fde974a7c45 Jul 31 22:57:58.858637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.858611 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pg6g5"] Jul 31 22:57:58.861276 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:57:58.861252 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e665f6f_530c_4b2b_967b_8d4be419d54a.slice/crio-99cbd14075e0deaec30391847878310bee7cdc044d7a43a7074821ee5d6b0ec3 WatchSource:0}: Error finding container 99cbd14075e0deaec30391847878310bee7cdc044d7a43a7074821ee5d6b0ec3: Status 404 returned error can't find the container with id 99cbd14075e0deaec30391847878310bee7cdc044d7a43a7074821ee5d6b0ec3 Jul 31 22:57:58.958515 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.958484 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pg6g5" event={"ID":"1e665f6f-530c-4b2b-967b-8d4be419d54a","Type":"ContainerStarted","Data":"99cbd14075e0deaec30391847878310bee7cdc044d7a43a7074821ee5d6b0ec3"} Jul 31 22:57:58.960063 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.960040 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvnqs" event={"ID":"3fb26301-9bdf-4865-a8fa-1e1a00d4246c","Type":"ContainerStarted","Data":"fde7cb4428f4ff1199c5620eb3b1601ead781648d7c73f83c2256b774faf806e"} Jul 31 22:57:58.960165 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.960105 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:57:58.961920 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.961896 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" event={"ID":"6fb1b3ae-60b2-4cb9-9499-38b7e750c488","Type":"ContainerStarted","Data":"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79"} Jul 31 22:57:58.962032 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:58.961943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" event={"ID":"6fb1b3ae-60b2-4cb9-9499-38b7e750c488","Type":"ContainerStarted","Data":"bb06e003963fb93b1c900e3dfd1606691a827b43d5f969f5232b0fde974a7c45"} Jul 31 22:57:59.966842 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:59.966526 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:57:59.994193 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:59.994138 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvnqs" podStartSLOduration=66.994119214 podStartE2EDuration="1m6.994119214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:58.981037038 +0000 UTC m=+66.903304810" watchObservedRunningTime="2026-07-31 22:57:59.994119214 +0000 UTC m=+67.916386985" Jul 31 22:57:59.994501 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:57:59.994473 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" podStartSLOduration=66.994463156 podStartE2EDuration="1m6.994463156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:57:59.991975687 +0000 UTC m=+67.914243457" watchObservedRunningTime="2026-07-31 22:57:59.994463156 +0000 UTC m=+67.916730925" Jul 31 22:58:02.977839 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:02.977785 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pg6g5" event={"ID":"1e665f6f-530c-4b2b-967b-8d4be419d54a","Type":"ContainerStarted","Data":"8b397056504b927b9710b7e93d39107a51f24721045450775f9b2d7fa0209779"} Jul 31 22:58:02.978219 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:02.977851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pg6g5" event={"ID":"1e665f6f-530c-4b2b-967b-8d4be419d54a","Type":"ContainerStarted","Data":"88da5f50b6853f9c0f6d7de638c245aced3c434e7b753ed21ef3d1cc35d7495a"} Jul 31 22:58:02.978219 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:02.977880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pg6g5" Jul 31 22:58:02.979234 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:02.979208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rrsk" event={"ID":"091a5632-061d-4b31-8e30-f03fffa6a68d","Type":"ContainerStarted","Data":"51073161e43b52b56a004afa30a00f0e05d5dfec0f3833caa5103abc63bfd47f"} Jul 31 22:58:02.998334 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:02.998289 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pg6g5" podStartSLOduration=33.66012623 podStartE2EDuration="36.998276266s" totalImagesPullingTime="3.338150036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:02.997817113 +0000 UTC m=+70.920084882" watchObservedRunningTime="2026-07-31 22:58:02.998276266 +0000 UTC m=+70.920544065" Jul 31 22:58:03.014515 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:03.014480 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2rrsk" podStartSLOduration=65.412366629 podStartE2EDuration="1m10.014467032s" totalImagesPullingTime="4.602100403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:03.014334846 +0000 UTC m=+70.936602617" watchObservedRunningTime="2026-07-31 22:58:03.014467032 +0000 UTC m=+70.936734801" Jul 31 22:58:04.330539 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:04.330511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:04.330887 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:04.330562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:04.330887 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:04.330647 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 22:58:04.330887 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:04.330689 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls podName:78518aa9-6d05-47ec-b30a-7c6fdb8f4c31 nodeName:}" failed. No retries permitted until 2026-07-31 22:58:20.330677514 +0000 UTC m=+88.252945263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mw7k9" (UID: "78518aa9-6d05-47ec-b30a-7c6fdb8f4c31") : secret "insights-runtime-extractor-tls" not found Jul 31 22:58:04.331013 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:04.330994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:07.532797 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.532766 2570 scope.go:117] "RemoveContainer" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" Jul 31 22:58:07.996667 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.996638 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 22:58:07.997097 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.997079 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/1.log" Jul 31 22:58:07.997182 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.997117 2570 generic.go:356] "Generic (PLEG): container finished" podID="78954292-2379-439c-babe-b86b2156e4a1" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" exitCode=255 Jul 31 22:58:07.997221 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.997178 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerDied","Data":"040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f"} Jul 31 22:58:07.997254 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.997237 2570 scope.go:117] "RemoveContainer" containerID="b0e6fb225f863b831f60f00a0f90d5bd5892538074ec2fbe965b284bab6320be" Jul 31 22:58:07.997534 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:07.997517 2570 scope.go:117] "RemoveContainer" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" Jul 31 22:58:07.997723 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:07.997704 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:58:09.001969 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:09.001945 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 22:58:12.984712 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:12.984677 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pg6g5" Jul 31 22:58:13.128990 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.128960 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:58:13.129246 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.129231 2570 scope.go:117] "RemoveContainer" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" Jul 31 22:58:13.129380 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:13.129362 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:58:13.524963 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.524937 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg"] Jul 31 22:58:13.527636 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.527619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.530604 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.530579 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6"] Jul 31 22:58:13.530801 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.530783 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 31 22:58:13.531149 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.531125 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-54mrh\"" Jul 31 22:58:13.531995 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.531977 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 22:58:13.532092 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.531979 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 22:58:13.533367 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.533350 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.536404 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.536384 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 31 22:58:13.540367 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.540348 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 22:58:13.549005 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.548951 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg"] Jul 31 22:58:13.561737 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.561717 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6"] Jul 31 22:58:13.586983 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.586964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee2a229b-3213-41ce-916f-5a46560d5d55-klusterlet-config\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.587082 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.586991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00879efc-90e7-4357-8601-d78036adffbf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.587082 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.587013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hmf\" (UniqueName: \"kubernetes.io/projected/00879efc-90e7-4357-8601-d78036adffbf-kube-api-access-x4hmf\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.587172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.587077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp6h\" (UniqueName: \"kubernetes.io/projected/ee2a229b-3213-41ce-916f-5a46560d5d55-kube-api-access-9cp6h\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.587172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.587126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee2a229b-3213-41ce-916f-5a46560d5d55-tmp\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.687927 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.687905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00879efc-90e7-4357-8601-d78036adffbf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.688017 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.687937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hmf\" (UniqueName: \"kubernetes.io/projected/00879efc-90e7-4357-8601-d78036adffbf-kube-api-access-x4hmf\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.688017 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.687969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp6h\" (UniqueName: \"kubernetes.io/projected/ee2a229b-3213-41ce-916f-5a46560d5d55-kube-api-access-9cp6h\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.688017 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.688010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee2a229b-3213-41ce-916f-5a46560d5d55-tmp\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.688123 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.688033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee2a229b-3213-41ce-916f-5a46560d5d55-klusterlet-config\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.688372 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.688353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee2a229b-3213-41ce-916f-5a46560d5d55-tmp\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.690263 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.690245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00879efc-90e7-4357-8601-d78036adffbf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.690263 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.690258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee2a229b-3213-41ce-916f-5a46560d5d55-klusterlet-config\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.697803 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.697781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hmf\" (UniqueName: \"kubernetes.io/projected/00879efc-90e7-4357-8601-d78036adffbf-kube-api-access-x4hmf\") pod \"managed-serviceaccount-addon-agent-77c789f84b-7t8zg\" (UID: \"00879efc-90e7-4357-8601-d78036adffbf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.697988 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.697972 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp6h\" (UniqueName: \"kubernetes.io/projected/ee2a229b-3213-41ce-916f-5a46560d5d55-kube-api-access-9cp6h\") pod \"klusterlet-addon-workmgr-5649df767d-994l6\" (UID: \"ee2a229b-3213-41ce-916f-5a46560d5d55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.844943 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.844899 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" Jul 31 22:58:13.850806 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.850792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:13.994080 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:13.994012 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg"] Jul 31 22:58:13.997190 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:13.997166 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00879efc_90e7_4357_8601_d78036adffbf.slice/crio-9c651e5c4e119070ccbb68aae977be705faaf71eb4924775844ac55ff333ae41 WatchSource:0}: Error finding container 9c651e5c4e119070ccbb68aae977be705faaf71eb4924775844ac55ff333ae41: Status 404 returned error can't find the container with id 9c651e5c4e119070ccbb68aae977be705faaf71eb4924775844ac55ff333ae41 Jul 31 22:58:14.012660 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:14.012643 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6"] Jul 31 22:58:14.014376 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:14.014352 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a229b_3213_41ce_916f_5a46560d5d55.slice/crio-704fa64aebe7db4283ac899efe639b097bbaec42c8e99446ae682027b57976c1 WatchSource:0}: Error finding container 704fa64aebe7db4283ac899efe639b097bbaec42c8e99446ae682027b57976c1: Status 404 returned error can't find the container with id 704fa64aebe7db4283ac899efe639b097bbaec42c8e99446ae682027b57976c1 Jul 31 22:58:14.019271 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:14.019247 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" event={"ID":"00879efc-90e7-4357-8601-d78036adffbf","Type":"ContainerStarted","Data":"9c651e5c4e119070ccbb68aae977be705faaf71eb4924775844ac55ff333ae41"} Jul 31 22:58:14.020237 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:14.020217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" event={"ID":"ee2a229b-3213-41ce-916f-5a46560d5d55","Type":"ContainerStarted","Data":"704fa64aebe7db4283ac899efe639b097bbaec42c8e99446ae682027b57976c1"} Jul 31 22:58:16.904436 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:16.904305 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:58:16.905026 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:16.904733 2570 scope.go:117] "RemoveContainer" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" Jul 31 22:58:16.905026 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:16.904946 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:58:18.034693 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.034654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" event={"ID":"00879efc-90e7-4357-8601-d78036adffbf","Type":"ContainerStarted","Data":"c0eb8360cfedf6d9743b2109a0842d793cff271bb5fde7c717861aeb213e86e6"} Jul 31 22:58:18.036103 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.036075 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" event={"ID":"ee2a229b-3213-41ce-916f-5a46560d5d55","Type":"ContainerStarted","Data":"c9be9a97b18684a6ee88d5c55ef29fec7b597227c03ea27074de0a8ad5612f62"} Jul 31 22:58:18.036255 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.036237 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:18.037679 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.037650 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" Jul 31 22:58:18.051663 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.051628 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77c789f84b-7t8zg" podStartSLOduration=1.204741371 podStartE2EDuration="5.051593743s" totalImagesPullingTime="3.846852372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:18.050497127 +0000 UTC m=+85.972764896" watchObservedRunningTime="2026-07-31 22:58:18.051593743 +0000 UTC m=+85.973861512" Jul 31 22:58:18.069644 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.069611 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5649df767d-994l6" podStartSLOduration=1.228371134 podStartE2EDuration="5.069603421s" totalImagesPullingTime="3.841232287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:18.068750374 +0000 UTC m=+85.991018159" watchObservedRunningTime="2026-07-31 22:58:18.069603421 +0000 UTC m=+85.991871189" Jul 31 22:58:18.660102 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.660067 2570 patch_prober.go:28] interesting pod/image-registry-7b56b88d89-tdg28 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 31 22:58:18.660276 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:18.660120 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 22:58:20.339985 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.339950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:20.342135 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.342112 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78518aa9-6d05-47ec-b30a-7c6fdb8f4c31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mw7k9\" (UID: \"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31\") " pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:20.351913 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.351889 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmjpj\"" Jul 31 22:58:20.360229 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.360210 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mw7k9" Jul 31 22:58:20.481920 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.481894 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mw7k9"] Jul 31 22:58:20.484649 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:20.484616 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78518aa9_6d05_47ec_b30a_7c6fdb8f4c31.slice/crio-28a839d0075842d58bb6a1ea092a0b460e7b6636ec0d575cf2bf0e8e4c2050e5 WatchSource:0}: Error finding container 28a839d0075842d58bb6a1ea092a0b460e7b6636ec0d575cf2bf0e8e4c2050e5: Status 404 returned error can't find the container with id 28a839d0075842d58bb6a1ea092a0b460e7b6636ec0d575cf2bf0e8e4c2050e5 Jul 31 22:58:20.974381 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:20.974349 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:58:21.047142 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:21.047104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mw7k9" event={"ID":"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31","Type":"ContainerStarted","Data":"e680f3155674487e995575370669fea4a202c13a847e0aba73c366b259cc667f"} Jul 31 22:58:21.047285 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:21.047167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mw7k9" event={"ID":"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31","Type":"ContainerStarted","Data":"28a839d0075842d58bb6a1ea092a0b460e7b6636ec0d575cf2bf0e8e4c2050e5"} Jul 31 22:58:22.052223 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:22.052190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mw7k9" event={"ID":"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31","Type":"ContainerStarted","Data":"3d74fefcf0eb612a825f2999ddf06d2a5bfec198d6011a96ead7e6cb45c67979"} Jul 31 22:58:23.057680 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:23.057643 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mw7k9" event={"ID":"78518aa9-6d05-47ec-b30a-7c6fdb8f4c31","Type":"ContainerStarted","Data":"a31dba87f0303f0803caea4211639f33c153eda920eedcd2f65a3e5f75502a37"} Jul 31 22:58:23.077867 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:23.077807 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mw7k9" podStartSLOduration=33.222488878 podStartE2EDuration="35.077794341s" totalImagesPullingTime="1.855305463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 22:57:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:23.077483162 +0000 UTC m=+90.999750931" watchObservedRunningTime="2026-07-31 22:58:23.077794341 +0000 UTC m=+91.000062111" Jul 31 22:58:26.617170 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.617139 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mc5xk"] Jul 31 22:58:26.620354 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.620337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.631874 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.631855 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 22:58:26.632004 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.631985 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 22:58:26.635677 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.634087 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 22:58:26.635677 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.635092 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 22:58:26.635677 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.635135 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 22:58:26.635677 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.635171 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgbfl\"" Jul 31 22:58:26.638084 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.636132 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 22:58:26.680133 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680241 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-sys\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680241 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-tls\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680241 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-root\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk28\" (UniqueName: \"kubernetes.io/projected/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-kube-api-access-9mk28\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-textfile\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-wtmp\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.680393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.680361 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-metrics-client-ca\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781449 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-tls\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781558 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-root\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781558 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk28\" (UniqueName: \"kubernetes.io/projected/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-kube-api-access-9mk28\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781558 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-textfile\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781558 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-wtmp\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781768 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-root\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781768 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781768 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-metrics-client-ca\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781768 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.781768 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-wtmp\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.782036 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-sys\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.782036 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.781922 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-sys\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.782036 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.782011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-textfile\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.782251 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.782231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.782308 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.782272 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-metrics-client-ca\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.783880 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.783861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.784078 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.784061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-node-exporter-tls\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.790946 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.790927 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk28\" (UniqueName: \"kubernetes.io/projected/0f1f8c22-a0c8-4777-b8f9-f5355a2bf892-kube-api-access-9mk28\") pod \"node-exporter-mc5xk\" (UID: \"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892\") " pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.929526 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:26.929468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc5xk" Jul 31 22:58:26.940420 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:26.940394 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1f8c22_a0c8_4777_b8f9_f5355a2bf892.slice/crio-635752e1959741b4be6c53bc98105ca5dfcacb1d972de6cb7b445274b85084c5 WatchSource:0}: Error finding container 635752e1959741b4be6c53bc98105ca5dfcacb1d972de6cb7b445274b85084c5: Status 404 returned error can't find the container with id 635752e1959741b4be6c53bc98105ca5dfcacb1d972de6cb7b445274b85084c5 Jul 31 22:58:27.071813 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:27.071786 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc5xk" event={"ID":"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892","Type":"ContainerStarted","Data":"635752e1959741b4be6c53bc98105ca5dfcacb1d972de6cb7b445274b85084c5"} Jul 31 22:58:27.533615 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:27.533585 2570 scope.go:117] "RemoveContainer" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" Jul 31 22:58:27.533830 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:58:27.533790 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hwdkz_openshift-console-operator(78954292-2379-439c-babe-b86b2156e4a1)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" podUID="78954292-2379-439c-babe-b86b2156e4a1" Jul 31 22:58:28.076175 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:28.076142 2570 generic.go:356] "Generic (PLEG): container finished" podID="0f1f8c22-a0c8-4777-b8f9-f5355a2bf892" containerID="5cc1e8b9587502f70cab4add3c48a576912a2004cde7ce65ebfaa2645cea5ad8" exitCode=0 Jul 31 22:58:28.076532 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:28.076190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc5xk" event={"ID":"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892","Type":"ContainerDied","Data":"5cc1e8b9587502f70cab4add3c48a576912a2004cde7ce65ebfaa2645cea5ad8"} Jul 31 22:58:29.082035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:29.081997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc5xk" event={"ID":"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892","Type":"ContainerStarted","Data":"a88c4796d520537768c751b9aa5403e97934604db5e038acde78d73345c0a44f"} Jul 31 22:58:29.082511 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:29.082051 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc5xk" event={"ID":"0f1f8c22-a0c8-4777-b8f9-f5355a2bf892","Type":"ContainerStarted","Data":"a57004ae275b49abb5eec336db7e1850eb94f028d6d24368156470b65bface92"} Jul 31 22:58:29.104764 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:29.104718 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mc5xk" podStartSLOduration=2.10189438 podStartE2EDuration="3.104701818s" totalImagesPullingTime="938.597058ms" totalInitContainerRuntime="64.21038ms" isStatefulPod=true podCreationTimestamp="2026-07-31 22:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:58:29.103771907 +0000 UTC m=+97.026039697" watchObservedRunningTime="2026-07-31 22:58:29.104701818 +0000 UTC m=+97.026969585" Jul 31 22:58:29.968903 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:29.968874 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvnqs" Jul 31 22:58:35.454241 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:35.454206 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:58:42.534272 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:42.534238 2570 scope.go:117] "RemoveContainer" containerID="040e627fcd65fcb86bee7ff8fed4758a697600ec4f3fd41487c5641f9d84be9f" Jul 31 22:58:43.129921 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.129893 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 22:58:43.130086 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.129986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" event={"ID":"78954292-2379-439c-babe-b86b2156e4a1","Type":"ContainerStarted","Data":"d45164e79dfb0e1af88ffd4c89e3aaa5f64860fdf2a2e87e3b8d6675f38d2217"} Jul 31 22:58:43.130285 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.130266 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:58:43.456670 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.456637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hwdkz" Jul 31 22:58:43.658908 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.658873 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-rgtg5"] Jul 31 22:58:43.661987 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.661968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:58:43.664574 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.664556 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 22:58:43.664653 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.664633 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 22:58:43.665068 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.665054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cx7xs\"" Jul 31 22:58:43.677214 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.677196 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-rgtg5"] Jul 31 22:58:43.794874 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.794792 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pwl\" (UniqueName: \"kubernetes.io/projected/a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9-kube-api-access-z8pwl\") pod \"downloads-77c974ffcf-rgtg5\" (UID: \"a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9\") " pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:58:43.895320 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.895287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pwl\" (UniqueName: \"kubernetes.io/projected/a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9-kube-api-access-z8pwl\") pod \"downloads-77c974ffcf-rgtg5\" (UID: \"a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9\") " pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:58:43.903035 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.903009 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pwl\" (UniqueName: \"kubernetes.io/projected/a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9-kube-api-access-z8pwl\") pod \"downloads-77c974ffcf-rgtg5\" (UID: \"a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9\") " pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:58:43.970073 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:43.970049 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:58:44.093674 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:44.093641 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-rgtg5"] Jul 31 22:58:44.094564 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:44.094534 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ed22fe_e2ba_4afe_a48b_4a4dafadfdd9.slice/crio-0eebef96e7420f1e6b2d0d5be389d01631de4ea352e6023a953a2f4ba6ea8365 WatchSource:0}: Error finding container 0eebef96e7420f1e6b2d0d5be389d01631de4ea352e6023a953a2f4ba6ea8365: Status 404 returned error can't find the container with id 0eebef96e7420f1e6b2d0d5be389d01631de4ea352e6023a953a2f4ba6ea8365 Jul 31 22:58:44.134075 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:44.134046 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-rgtg5" event={"ID":"a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9","Type":"ContainerStarted","Data":"0eebef96e7420f1e6b2d0d5be389d01631de4ea352e6023a953a2f4ba6ea8365"} Jul 31 22:58:53.493556 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.493517 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:58:53.528019 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.527988 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:58:53.528174 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.528128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.533190 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.533166 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 22:58:53.533326 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.533165 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 22:58:53.533961 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.533929 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 22:58:53.534092 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.533966 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 22:58:53.534092 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.534027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-chht8\"" Jul 31 22:58:53.534196 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.534167 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 22:58:53.679580 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.679767 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679599 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64m8k\" (UniqueName: \"kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.679767 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.679767 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679672 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.679767 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.680012 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.679783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.780966 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.780884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.780966 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.780929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.780995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.781036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64m8k\" (UniqueName: \"kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.781064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781172 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.781101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781708 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.781678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.781844 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.781681 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.782077 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.782046 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.783793 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.783767 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.784167 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.784136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.794309 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.794284 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64m8k\" (UniqueName: \"kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k\") pod \"console-79f8d5dc96-xtvbr\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:53.838449 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:53.838425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:58:59.559700 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:58:59.559677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:58:59.561704 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:58:59.561672 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a4454e_9a07_4c20_ae76_9f19f96ac233.slice/crio-0da873fe4d123ac0003baf7b4de38360223390cd0ca8ebca492fc72523822081 WatchSource:0}: Error finding container 0da873fe4d123ac0003baf7b4de38360223390cd0ca8ebca492fc72523822081: Status 404 returned error can't find the container with id 0da873fe4d123ac0003baf7b4de38360223390cd0ca8ebca492fc72523822081 Jul 31 22:59:00.199457 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.199411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-rgtg5" event={"ID":"a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9","Type":"ContainerStarted","Data":"0bd8a2c27ed76642f8596f0b07a710ad3a4089972ab8efe2b2d6c612653767b7"} Jul 31 22:59:00.199874 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.199834 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:59:00.202121 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.202091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8d5dc96-xtvbr" event={"ID":"52a4454e-9a07-4c20-ae76-9f19f96ac233","Type":"ContainerStarted","Data":"0da873fe4d123ac0003baf7b4de38360223390cd0ca8ebca492fc72523822081"} Jul 31 22:59:00.210230 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.210207 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-rgtg5" Jul 31 22:59:00.221615 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.221566 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-rgtg5" podStartSLOduration=1.804833258 podStartE2EDuration="17.221549342s" totalImagesPullingTime="15.416716084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:59:00.219455887 +0000 UTC m=+128.141723668" watchObservedRunningTime="2026-07-31 22:59:00.221549342 +0000 UTC m=+128.143817112" Jul 31 22:59:00.473124 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.472898 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerName="registry" containerID="cri-o://0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79" gracePeriod=30 Jul 31 22:59:00.752776 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.752750 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:59:00.846292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846189 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmxvk\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846232 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846292 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846270 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846317 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846360 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846408 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.846637 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.846439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") pod \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\" (UID: \"6fb1b3ae-60b2-4cb9-9499-38b7e750c488\") " Jul 31 22:59:00.847178 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.847147 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 22:59:00.847725 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.847656 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 22:59:00.850025 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.849997 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 22:59:00.850385 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.850356 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk" (OuterVolumeSpecName: "kube-api-access-mmxvk") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "kube-api-access-mmxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 22:59:00.850553 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.850522 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 22:59:00.851298 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.851227 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 22:59:00.853260 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.853167 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 22:59:00.859481 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.859453 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6fb1b3ae-60b2-4cb9-9499-38b7e750c488" (UID: "6fb1b3ae-60b2-4cb9-9499-38b7e750c488"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 22:59:00.948200 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948156 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-tls\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948200 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948198 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-registry-certificates\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948214 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxvk\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-kube-api-access-mmxvk\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948230 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-trusted-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948240 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-installation-pull-secrets\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948248 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-ca-trust-extracted\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948258 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-bound-sa-token\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:00.948480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:00.948272 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fb1b3ae-60b2-4cb9-9499-38b7e750c488-image-registry-private-configuration\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:01.208175 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.208132 2570 generic.go:356] "Generic (PLEG): container finished" podID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerID="0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79" exitCode=0 Jul 31 22:59:01.208961 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.208550 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" Jul 31 22:59:01.208961 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.208705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" event={"ID":"6fb1b3ae-60b2-4cb9-9499-38b7e750c488","Type":"ContainerDied","Data":"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79"} Jul 31 22:59:01.208961 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.208754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b56b88d89-tdg28" event={"ID":"6fb1b3ae-60b2-4cb9-9499-38b7e750c488","Type":"ContainerDied","Data":"bb06e003963fb93b1c900e3dfd1606691a827b43d5f969f5232b0fde974a7c45"} Jul 31 22:59:01.208961 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.208780 2570 scope.go:117] "RemoveContainer" containerID="0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79" Jul 31 22:59:01.220607 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.220488 2570 scope.go:117] "RemoveContainer" containerID="0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79" Jul 31 22:59:01.220979 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:59:01.220950 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79\": container with ID starting with 0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79 not found: ID does not exist" containerID="0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79" Jul 31 22:59:01.221072 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.220991 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79"} err="failed to get container status \"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79\": rpc error: code = NotFound desc = could not find container \"0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79\": container with ID starting with 0f12901a778f0d69635b87b832ed2a461042f37a8942966551ee87e39b341a79 not found: ID does not exist" Jul 31 22:59:01.235029 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.234984 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:59:01.241300 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.241278 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b56b88d89-tdg28"] Jul 31 22:59:01.452550 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.452516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:59:01.455131 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.455101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/842f999d-a623-4c09-8218-099a42fe84cc-metrics-certs\") pod \"network-metrics-daemon-9ftrs\" (UID: \"842f999d-a623-4c09-8218-099a42fe84cc\") " pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:59:01.748096 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.748059 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8t45\"" Jul 31 22:59:01.755699 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.755595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ftrs" Jul 31 22:59:01.948262 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.947814 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 22:59:01.948262 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.948195 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerName="registry" Jul 31 22:59:01.948262 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.948213 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerName="registry" Jul 31 22:59:01.948529 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.948280 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" containerName="registry" Jul 31 22:59:01.965964 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.965930 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 22:59:01.966123 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.966057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:01.975082 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:01.974896 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 22:59:02.058684 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.058926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.058926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.058926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.058926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.058926 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.059184 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.058951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzh4\" (UniqueName: \"kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160118 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160118 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160352 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160352 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160352 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzh4\" (UniqueName: \"kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160352 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160250 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.160352 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.161232 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.160893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.161887 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.161862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.162393 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.162366 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.163533 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.163514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.163984 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.163960 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.164151 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.164107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.170910 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.170867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzh4\" (UniqueName: \"kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4\") pod \"console-7b45fd4655-z8k2g\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.280000 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.279964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:02.539673 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.539641 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb1b3ae-60b2-4cb9-9499-38b7e750c488" path="/var/lib/kubelet/pods/6fb1b3ae-60b2-4cb9-9499-38b7e750c488/volumes" Jul 31 22:59:02.888934 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.888508 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 22:59:02.891493 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:59:02.891461 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165bc3a4_0f27_4632_8bfa_6c2a9d6f4e57.slice/crio-089315ff3bd12a48c4c94b417e13def46ad11c60305073d236cdd42cadf6e646 WatchSource:0}: Error finding container 089315ff3bd12a48c4c94b417e13def46ad11c60305073d236cdd42cadf6e646: Status 404 returned error can't find the container with id 089315ff3bd12a48c4c94b417e13def46ad11c60305073d236cdd42cadf6e646 Jul 31 22:59:02.902549 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:02.902505 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9ftrs"] Jul 31 22:59:02.905287 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:59:02.905258 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842f999d_a623_4c09_8218_099a42fe84cc.slice/crio-eebfd7547b2d74feaaa5e99f83a8d87bca60a7ada7b4bd90abf512109dbc4b53 WatchSource:0}: Error finding container eebfd7547b2d74feaaa5e99f83a8d87bca60a7ada7b4bd90abf512109dbc4b53: Status 404 returned error can't find the container with id eebfd7547b2d74feaaa5e99f83a8d87bca60a7ada7b4bd90abf512109dbc4b53 Jul 31 22:59:03.220099 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:03.220073 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b45fd4655-z8k2g" event={"ID":"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57","Type":"ContainerStarted","Data":"089315ff3bd12a48c4c94b417e13def46ad11c60305073d236cdd42cadf6e646"} Jul 31 22:59:03.221395 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:03.221366 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ftrs" event={"ID":"842f999d-a623-4c09-8218-099a42fe84cc","Type":"ContainerStarted","Data":"eebfd7547b2d74feaaa5e99f83a8d87bca60a7ada7b4bd90abf512109dbc4b53"} Jul 31 22:59:04.228927 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:04.228815 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8d5dc96-xtvbr" event={"ID":"52a4454e-9a07-4c20-ae76-9f19f96ac233","Type":"ContainerStarted","Data":"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080"} Jul 31 22:59:04.231462 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:04.231383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b45fd4655-z8k2g" event={"ID":"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57","Type":"ContainerStarted","Data":"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5"} Jul 31 22:59:04.250370 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:04.250311 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f8d5dc96-xtvbr" podStartSLOduration=7.643771889 podStartE2EDuration="11.250289642s" totalImagesPullingTime="3.606517753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:59:04.249204868 +0000 UTC m=+132.171472636" watchObservedRunningTime="2026-07-31 22:59:04.250289642 +0000 UTC m=+132.172557414" Jul 31 22:59:04.282585 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:04.282544 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b45fd4655-z8k2g" podStartSLOduration=2.7079311390000003 podStartE2EDuration="3.28252969s" totalImagesPullingTime="574.598551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:59:04.280975803 +0000 UTC m=+132.203243594" watchObservedRunningTime="2026-07-31 22:59:04.28252969 +0000 UTC m=+132.204797458" Jul 31 22:59:05.237339 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:05.237296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ftrs" event={"ID":"842f999d-a623-4c09-8218-099a42fe84cc","Type":"ContainerStarted","Data":"30ce7f66747bde504ecf4b7f5bb3867ef2285c40d435c5646b6730af8fcf894c"} Jul 31 22:59:05.237881 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:05.237362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ftrs" event={"ID":"842f999d-a623-4c09-8218-099a42fe84cc","Type":"ContainerStarted","Data":"b85d977bf30f850805095db4451f45570cfcbea614b112199de163872780ec9a"} Jul 31 22:59:05.255602 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:05.255558 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9ftrs" podStartSLOduration=130.488897953 podStartE2EDuration="2m12.255542451s" totalImagesPullingTime="1.766644498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:59:05.253801283 +0000 UTC m=+133.176069044" watchObservedRunningTime="2026-07-31 22:59:05.255542451 +0000 UTC m=+133.177810245" Jul 31 22:59:12.280764 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:12.280724 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:12.281309 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:12.280791 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:12.285261 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:12.285237 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:13.272773 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:13.272739 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 22:59:13.325230 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:13.325194 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:59:13.838982 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:13.838953 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:59:38.345369 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.345299 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f8d5dc96-xtvbr" podUID="52a4454e-9a07-4c20-ae76-9f19f96ac233" containerName="console" containerID="cri-o://0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080" gracePeriod=15 Jul 31 22:59:38.602150 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.602090 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:59:38.711343 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711314 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711456 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711350 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711456 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711371 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711456 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711398 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711564 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711509 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64m8k\" (UniqueName: \"kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711564 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711541 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config\") pod \"52a4454e-9a07-4c20-ae76-9f19f96ac233\" (UID: \"52a4454e-9a07-4c20-ae76-9f19f96ac233\") " Jul 31 22:59:38.711787 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711750 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca" (OuterVolumeSpecName: "service-ca") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 22:59:38.711787 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711758 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config" (OuterVolumeSpecName: "console-config") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 22:59:38.711787 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.711746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 22:59:38.713550 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.713518 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 22:59:38.713643 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.713606 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 22:59:38.713643 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.713617 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k" (OuterVolumeSpecName: "kube-api-access-64m8k") pod "52a4454e-9a07-4c20-ae76-9f19f96ac233" (UID: "52a4454e-9a07-4c20-ae76-9f19f96ac233"). InnerVolumeSpecName "kube-api-access-64m8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 22:59:38.812097 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812073 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:38.812188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812099 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:38.812188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812112 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64m8k\" (UniqueName: \"kubernetes.io/projected/52a4454e-9a07-4c20-ae76-9f19f96ac233-kube-api-access-64m8k\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:38.812188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812124 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-oauth-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:38.812188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812133 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-oauth-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:38.812188 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:38.812141 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52a4454e-9a07-4c20-ae76-9f19f96ac233-console-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 22:59:39.363045 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.363006 2570 generic.go:356] "Generic (PLEG): container finished" podID="52a4454e-9a07-4c20-ae76-9f19f96ac233" containerID="0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080" exitCode=0 Jul 31 22:59:39.363480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.363062 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8d5dc96-xtvbr" Jul 31 22:59:39.363480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.363090 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8d5dc96-xtvbr" event={"ID":"52a4454e-9a07-4c20-ae76-9f19f96ac233","Type":"ContainerDied","Data":"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080"} Jul 31 22:59:39.363480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.363136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8d5dc96-xtvbr" event={"ID":"52a4454e-9a07-4c20-ae76-9f19f96ac233","Type":"ContainerDied","Data":"0da873fe4d123ac0003baf7b4de38360223390cd0ca8ebca492fc72523822081"} Jul 31 22:59:39.363480 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.363158 2570 scope.go:117] "RemoveContainer" containerID="0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080" Jul 31 22:59:39.378191 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.378171 2570 scope.go:117] "RemoveContainer" containerID="0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080" Jul 31 22:59:39.378503 ip-10-0-137-58 kubenswrapper[2570]: E0731 22:59:39.378481 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080\": container with ID starting with 0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080 not found: ID does not exist" containerID="0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080" Jul 31 22:59:39.378548 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.378513 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080"} err="failed to get container status \"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080\": rpc error: code = NotFound desc = could not find container \"0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080\": container with ID starting with 0d51721af1bb04dc19e40713b806ad8b058fbec10369d9c6a8a60506157b4080 not found: ID does not exist" Jul 31 22:59:39.384272 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.384243 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:59:39.387905 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:39.387877 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f8d5dc96-xtvbr"] Jul 31 22:59:40.537469 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:40.537424 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a4454e-9a07-4c20-ae76-9f19f96ac233" path="/var/lib/kubelet/pods/52a4454e-9a07-4c20-ae76-9f19f96ac233/volumes" Jul 31 22:59:54.110786 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.110742 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 22:59:54.111771 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.111741 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a4454e-9a07-4c20-ae76-9f19f96ac233" containerName="console" Jul 31 22:59:54.111771 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.111770 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a4454e-9a07-4c20-ae76-9f19f96ac233" containerName="console" Jul 31 22:59:54.111939 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.111862 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a4454e-9a07-4c20-ae76-9f19f96ac233" containerName="console" Jul 31 22:59:54.114645 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.114623 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.126111 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.126076 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 22:59:54.210474 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210604 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210604 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210720 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210599 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210720 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210720 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.210720 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.210713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcs5\" (UniqueName: \"kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.311742 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.311715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.311854 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.311752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcs5\" (UniqueName: \"kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.311854 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.311784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.311992 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.311974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312049 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312121 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312174 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312596 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312692 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312621 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312868 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.312969 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.312952 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.314333 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.314303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.314457 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.314437 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.320470 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.320451 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcs5\" (UniqueName: \"kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5\") pod \"console-846897d9c5-k2qt8\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.428982 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.428932 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 22:59:54.546502 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:54.546476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 22:59:54.548700 ip-10-0-137-58 kubenswrapper[2570]: W0731 22:59:54.548672 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ac6a56_a6ef_4799_acdc_37709047bfff.slice/crio-bea23225756c2aa2ba0e20399cb27b01f0e30157f31a1fed845b1ee6ef9f06c1 WatchSource:0}: Error finding container bea23225756c2aa2ba0e20399cb27b01f0e30157f31a1fed845b1ee6ef9f06c1: Status 404 returned error can't find the container with id bea23225756c2aa2ba0e20399cb27b01f0e30157f31a1fed845b1ee6ef9f06c1 Jul 31 22:59:55.423028 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:55.422995 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846897d9c5-k2qt8" event={"ID":"16ac6a56-a6ef-4799-acdc-37709047bfff","Type":"ContainerStarted","Data":"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89"} Jul 31 22:59:55.423402 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:55.423045 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846897d9c5-k2qt8" event={"ID":"16ac6a56-a6ef-4799-acdc-37709047bfff","Type":"ContainerStarted","Data":"bea23225756c2aa2ba0e20399cb27b01f0e30157f31a1fed845b1ee6ef9f06c1"} Jul 31 22:59:55.444236 ip-10-0-137-58 kubenswrapper[2570]: I0731 22:59:55.444191 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846897d9c5-k2qt8" podStartSLOduration=1.444177114 podStartE2EDuration="1.444177114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 22:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 22:59:55.443064893 +0000 UTC m=+183.365332655" watchObservedRunningTime="2026-07-31 22:59:55.444177114 +0000 UTC m=+183.366444883" Jul 31 23:00:04.429245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:04.429199 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:00:04.429778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:04.429268 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:00:04.433969 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:04.433945 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:00:04.459061 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:04.459036 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:00:04.510137 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:04.510105 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 23:00:29.529169 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.529115 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b45fd4655-z8k2g" podUID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" containerName="console" containerID="cri-o://b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5" gracePeriod=15 Jul 31 23:00:29.767476 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.767454 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 23:00:29.949012 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.948983 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949167 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949028 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949167 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949079 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949167 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949103 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949167 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949125 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949167 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949147 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dzh4\" (UniqueName: \"kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949414 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949179 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config\") pod \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\" (UID: \"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57\") " Jul 31 23:00:29.949642 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949598 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca" (OuterVolumeSpecName: "service-ca") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:00:29.949642 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949608 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:00:29.949642 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949603 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:00:29.949857 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.949703 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config" (OuterVolumeSpecName: "console-config") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:00:29.950946 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.950918 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:00:29.951472 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.951449 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4" (OuterVolumeSpecName: "kube-api-access-8dzh4") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "kube-api-access-8dzh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:00:29.951472 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:29.951461 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" (UID: "165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:00:30.050111 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050093 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-oauth-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050112 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050122 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzh4\" (UniqueName: \"kubernetes.io/projected/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-kube-api-access-8dzh4\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050130 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050139 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-console-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050148 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-trusted-ca-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.050176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.050155 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57-oauth-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:00:30.547035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.547005 2570 generic.go:356] "Generic (PLEG): container finished" podID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" containerID="b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5" exitCode=0 Jul 31 23:00:30.547397 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.547061 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b45fd4655-z8k2g" Jul 31 23:00:30.547397 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.547080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b45fd4655-z8k2g" event={"ID":"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57","Type":"ContainerDied","Data":"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5"} Jul 31 23:00:30.547397 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.547126 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b45fd4655-z8k2g" event={"ID":"165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57","Type":"ContainerDied","Data":"089315ff3bd12a48c4c94b417e13def46ad11c60305073d236cdd42cadf6e646"} Jul 31 23:00:30.547397 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.547145 2570 scope.go:117] "RemoveContainer" containerID="b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5" Jul 31 23:00:30.554349 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.554334 2570 scope.go:117] "RemoveContainer" containerID="b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5" Jul 31 23:00:30.554640 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:00:30.554621 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5\": container with ID starting with b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5 not found: ID does not exist" containerID="b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5" Jul 31 23:00:30.554713 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.554657 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5"} err="failed to get container status \"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5\": rpc error: code = NotFound desc = could not find container \"b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5\": container with ID starting with b71550964b87465c736fc1d0835af3e41e9b2616ecf675ebaf5a77de80649fe5 not found: ID does not exist" Jul 31 23:00:30.564807 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.564786 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 23:00:30.568861 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:30.568840 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b45fd4655-z8k2g"] Jul 31 23:00:32.536659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:00:32.536620 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" path="/var/lib/kubelet/pods/165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57/volumes" Jul 31 23:01:12.123754 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.123722 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:01:12.124180 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.124077 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" containerName="console" Jul 31 23:01:12.124180 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.124098 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" containerName="console" Jul 31 23:01:12.124180 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.124153 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="165bc3a4-0f27-4632-8bfa-6c2a9d6f4e57" containerName="console" Jul 31 23:01:12.127073 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.127043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.142080 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.142055 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:01:12.204641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5m8c\" (UniqueName: \"kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204752 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204752 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204752 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204898 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204898 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.204962 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.204897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305385 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305360 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305385 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305500 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305562 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305610 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5m8c\" (UniqueName: \"kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305643 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.305675 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.305662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.306332 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.306299 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.306424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.306329 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.306424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.306375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.306544 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.306457 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.307975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.307943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.308173 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.308153 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.314461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.314439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5m8c\" (UniqueName: \"kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c\") pod \"console-6649ff6db9-7xf5q\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.436684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.436668 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:12.563222 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.563190 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:01:12.565916 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:01:12.565891 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eba0df4_71d2_4eee_a837_83157481d4ab.slice/crio-cbe71356a9f33155e67b76f9adc48a2ab69637658f8da6448add864db57cb946 WatchSource:0}: Error finding container cbe71356a9f33155e67b76f9adc48a2ab69637658f8da6448add864db57cb946: Status 404 returned error can't find the container with id cbe71356a9f33155e67b76f9adc48a2ab69637658f8da6448add864db57cb946 Jul 31 23:01:12.692138 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.692055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649ff6db9-7xf5q" event={"ID":"1eba0df4-71d2-4eee-a837-83157481d4ab","Type":"ContainerStarted","Data":"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e"} Jul 31 23:01:12.692138 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.692110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649ff6db9-7xf5q" event={"ID":"1eba0df4-71d2-4eee-a837-83157481d4ab","Type":"ContainerStarted","Data":"cbe71356a9f33155e67b76f9adc48a2ab69637658f8da6448add864db57cb946"} Jul 31 23:01:12.710730 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:12.710686 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6649ff6db9-7xf5q" podStartSLOduration=0.710671994 podStartE2EDuration="710.671994ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:01:12.70966775 +0000 UTC m=+260.631935540" watchObservedRunningTime="2026-07-31 23:01:12.710671994 +0000 UTC m=+260.632939801" Jul 31 23:01:22.436864 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:22.436831 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:22.437234 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:22.436886 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:22.441499 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:22.441477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:22.730572 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:22.730481 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:01:22.780418 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:22.780382 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 23:01:47.803976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:47.803928 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-846897d9c5-k2qt8" podUID="16ac6a56-a6ef-4799-acdc-37709047bfff" containerName="console" containerID="cri-o://f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89" gracePeriod=15 Jul 31 23:01:48.039588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.039557 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:01:48.151598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151481 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151549 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151593 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151912 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151624 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151912 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151658 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151912 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151688 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlcs5\" (UniqueName: \"kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.151912 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.151751 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config\") pod \"16ac6a56-a6ef-4799-acdc-37709047bfff\" (UID: \"16ac6a56-a6ef-4799-acdc-37709047bfff\") " Jul 31 23:01:48.152116 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.152052 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:01:48.152224 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.152199 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config" (OuterVolumeSpecName: "console-config") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:01:48.152304 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.152197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:01:48.152304 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.152230 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca" (OuterVolumeSpecName: "service-ca") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:01:48.153812 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.153788 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:01:48.153812 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.153804 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:01:48.153982 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.153788 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5" (OuterVolumeSpecName: "kube-api-access-tlcs5") pod "16ac6a56-a6ef-4799-acdc-37709047bfff" (UID: "16ac6a56-a6ef-4799-acdc-37709047bfff"). InnerVolumeSpecName "kube-api-access-tlcs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:01:48.252461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252413 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-trusted-ca-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252454 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252464 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252473 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-console-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252482 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16ac6a56-a6ef-4799-acdc-37709047bfff-oauth-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252731 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252490 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcs5\" (UniqueName: \"kubernetes.io/projected/16ac6a56-a6ef-4799-acdc-37709047bfff-kube-api-access-tlcs5\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.252731 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.252498 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16ac6a56-a6ef-4799-acdc-37709047bfff-console-oauth-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:01:48.822272 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.822182 2570 generic.go:356] "Generic (PLEG): container finished" podID="16ac6a56-a6ef-4799-acdc-37709047bfff" containerID="f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89" exitCode=0 Jul 31 23:01:48.822272 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.822225 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846897d9c5-k2qt8" Jul 31 23:01:48.822734 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.822277 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846897d9c5-k2qt8" event={"ID":"16ac6a56-a6ef-4799-acdc-37709047bfff","Type":"ContainerDied","Data":"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89"} Jul 31 23:01:48.822734 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.822340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846897d9c5-k2qt8" event={"ID":"16ac6a56-a6ef-4799-acdc-37709047bfff","Type":"ContainerDied","Data":"bea23225756c2aa2ba0e20399cb27b01f0e30157f31a1fed845b1ee6ef9f06c1"} Jul 31 23:01:48.822734 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.822373 2570 scope.go:117] "RemoveContainer" containerID="f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89" Jul 31 23:01:48.832207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.832190 2570 scope.go:117] "RemoveContainer" containerID="f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89" Jul 31 23:01:48.832464 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:01:48.832444 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89\": container with ID starting with f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89 not found: ID does not exist" containerID="f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89" Jul 31 23:01:48.832522 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.832476 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89"} err="failed to get container status \"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89\": rpc error: code = NotFound desc = could not find container \"f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89\": container with ID starting with f265fc31eeb8b9deb4704f206d0c327585ef02fa272a965b2c3c73f5dd6aba89 not found: ID does not exist" Jul 31 23:01:48.839312 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.839285 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 23:01:48.841481 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:48.841456 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846897d9c5-k2qt8"] Jul 31 23:01:50.537368 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:50.537335 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ac6a56-a6ef-4799-acdc-37709047bfff" path="/var/lib/kubelet/pods/16ac6a56-a6ef-4799-acdc-37709047bfff/volumes" Jul 31 23:01:52.446039 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:52.446006 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:01:52.446902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:52.446874 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:01:52.450741 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:52.450720 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:01:52.451452 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:52.451429 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:01:52.456681 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:01:52.456661 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 23:02:01.310872 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.310838 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5"] Jul 31 23:02:01.312245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.311087 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ac6a56-a6ef-4799-acdc-37709047bfff" containerName="console" Jul 31 23:02:01.312245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.311097 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ac6a56-a6ef-4799-acdc-37709047bfff" containerName="console" Jul 31 23:02:01.312245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.311169 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ac6a56-a6ef-4799-acdc-37709047bfff" containerName="console" Jul 31 23:02:01.313008 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.312992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.315927 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.315900 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 23:02:01.316875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.316855 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 23:02:01.316875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.316866 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mmk8v\"" Jul 31 23:02:01.326175 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.326155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5"] Jul 31 23:02:01.441302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.441265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.441302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.441304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjl9\" (UniqueName: \"kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.441518 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.441369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.542307 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.542277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.542442 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.542326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.542442 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.542345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjl9\" (UniqueName: \"kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.542654 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.542623 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.542715 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.542697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.551174 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.551144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjl9\" (UniqueName: \"kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.622155 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.622096 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:01.742375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.742351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5"] Jul 31 23:02:01.745169 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:01.745144 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81a0d58_589a_49ac_97a5_f10825ecb1ff.slice/crio-a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13 WatchSource:0}: Error finding container a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13: Status 404 returned error can't find the container with id a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13 Jul 31 23:02:01.747045 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.747025 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:02:01.870633 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:01.870599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" event={"ID":"a81a0d58-589a-49ac-97a5-f10825ecb1ff","Type":"ContainerStarted","Data":"a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13"} Jul 31 23:02:07.899780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:07.899740 2570 generic.go:356] "Generic (PLEG): container finished" podID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerID="6f7294afe36c76c32c49ebbd134e5fcd450725e6400d6647cfd3404d5fca6f08" exitCode=0 Jul 31 23:02:07.900264 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:07.899798 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" event={"ID":"a81a0d58-589a-49ac-97a5-f10825ecb1ff","Type":"ContainerDied","Data":"6f7294afe36c76c32c49ebbd134e5fcd450725e6400d6647cfd3404d5fca6f08"} Jul 31 23:02:10.912966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:10.912929 2570 generic.go:356] "Generic (PLEG): container finished" podID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerID="c139ac05ea167829ec8333d384cffec96c8ab6a6c4881e4e458c30f1b993c079" exitCode=0 Jul 31 23:02:10.913329 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:10.913011 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" event={"ID":"a81a0d58-589a-49ac-97a5-f10825ecb1ff","Type":"ContainerDied","Data":"c139ac05ea167829ec8333d384cffec96c8ab6a6c4881e4e458c30f1b993c079"} Jul 31 23:02:17.942858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:17.942801 2570 generic.go:356] "Generic (PLEG): container finished" podID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerID="72d3c6f79b8635e4efba63a49e2665164e907e79bfaeb1bf3caf922d695f274b" exitCode=0 Jul 31 23:02:17.943229 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:17.942884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" event={"ID":"a81a0d58-589a-49ac-97a5-f10825ecb1ff","Type":"ContainerDied","Data":"72d3c6f79b8635e4efba63a49e2665164e907e79bfaeb1bf3caf922d695f274b"} Jul 31 23:02:19.068158 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.068133 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:19.179608 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.179581 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util\") pod \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " Jul 31 23:02:19.179735 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.179619 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle\") pod \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " Jul 31 23:02:19.179735 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.179684 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwjl9\" (UniqueName: \"kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9\") pod \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\" (UID: \"a81a0d58-589a-49ac-97a5-f10825ecb1ff\") " Jul 31 23:02:19.180286 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.180259 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle" (OuterVolumeSpecName: "bundle") pod "a81a0d58-589a-49ac-97a5-f10825ecb1ff" (UID: "a81a0d58-589a-49ac-97a5-f10825ecb1ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:19.181835 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.181792 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9" (OuterVolumeSpecName: "kube-api-access-mwjl9") pod "a81a0d58-589a-49ac-97a5-f10825ecb1ff" (UID: "a81a0d58-589a-49ac-97a5-f10825ecb1ff"). InnerVolumeSpecName "kube-api-access-mwjl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:02:19.183699 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.183680 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util" (OuterVolumeSpecName: "util") pod "a81a0d58-589a-49ac-97a5-f10825ecb1ff" (UID: "a81a0d58-589a-49ac-97a5-f10825ecb1ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:19.280727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.280667 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:19.280727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.280691 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a81a0d58-589a-49ac-97a5-f10825ecb1ff-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:19.280727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.280700 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjl9\" (UniqueName: \"kubernetes.io/projected/a81a0d58-589a-49ac-97a5-f10825ecb1ff-kube-api-access-mwjl9\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:19.955118 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.955092 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" Jul 31 23:02:19.955266 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.955090 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4w5" event={"ID":"a81a0d58-589a-49ac-97a5-f10825ecb1ff","Type":"ContainerDied","Data":"a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13"} Jul 31 23:02:19.955266 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:19.955252 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1068778d03cc96d03140c47e07fdb79c14385e7430643622adce3951c133b13" Jul 31 23:02:25.075835 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.075792 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc"] Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076052 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="pull" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076063 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="pull" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076077 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="extract" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076083 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="extract" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076089 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="util" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076095 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="util" Jul 31 23:02:25.076178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.076142 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81a0d58-589a-49ac-97a5-f10825ecb1ff" containerName="extract" Jul 31 23:02:25.110507 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.110487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.112123 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.112090 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc"] Jul 31 23:02:25.114816 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.114789 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-822jz\"" Jul 31 23:02:25.115110 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.115089 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 31 23:02:25.115350 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.115336 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 31 23:02:25.224627 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.224601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4db1240-6346-40aa-b192-964faa093eaf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.224730 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.224642 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9gn\" (UniqueName: \"kubernetes.io/projected/a4db1240-6346-40aa-b192-964faa093eaf-kube-api-access-bx9gn\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.324826 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.324804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4db1240-6346-40aa-b192-964faa093eaf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.324887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.324859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9gn\" (UniqueName: \"kubernetes.io/projected/a4db1240-6346-40aa-b192-964faa093eaf-kube-api-access-bx9gn\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.325132 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.325116 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4db1240-6346-40aa-b192-964faa093eaf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.344229 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.344173 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9gn\" (UniqueName: \"kubernetes.io/projected/a4db1240-6346-40aa-b192-964faa093eaf-kube-api-access-bx9gn\") pod \"cert-manager-operator-controller-manager-55c444bf86-wrmkc\" (UID: \"a4db1240-6346-40aa-b192-964faa093eaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.420210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.420190 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" Jul 31 23:02:25.592770 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.592698 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc"] Jul 31 23:02:25.595008 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:25.594952 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4db1240_6346_40aa_b192_964faa093eaf.slice/crio-50e0ff98256b7052a4f0e3ed31e04afc87b9645e967ead0360db3bdfa0851789 WatchSource:0}: Error finding container 50e0ff98256b7052a4f0e3ed31e04afc87b9645e967ead0360db3bdfa0851789: Status 404 returned error can't find the container with id 50e0ff98256b7052a4f0e3ed31e04afc87b9645e967ead0360db3bdfa0851789 Jul 31 23:02:25.976750 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:25.976718 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" event={"ID":"a4db1240-6346-40aa-b192-964faa093eaf","Type":"ContainerStarted","Data":"50e0ff98256b7052a4f0e3ed31e04afc87b9645e967ead0360db3bdfa0851789"} Jul 31 23:02:27.985588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:27.985550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" event={"ID":"a4db1240-6346-40aa-b192-964faa093eaf","Type":"ContainerStarted","Data":"275a87e57fe9258a77b54d55da5061d29d7dc277118607c0954e7cfa5ed45b56"} Jul 31 23:02:28.007950 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.007898 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wrmkc" podStartSLOduration=0.956670991 podStartE2EDuration="3.007877851s" totalImagesPullingTime="2.05120686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:02:28.006180815 +0000 UTC m=+335.928448585" watchObservedRunningTime="2026-07-31 23:02:28.007877851 +0000 UTC m=+335.930145623" Jul 31 23:02:28.442329 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.442298 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vjtk4"] Jul 31 23:02:28.464545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.464513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vjtk4"] Jul 31 23:02:28.464675 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.464627 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.467624 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.467593 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 23:02:28.467997 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.467975 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 23:02:28.549998 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.549977 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.550109 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.550029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdrh\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-kube-api-access-6wdrh\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.613574 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.613549 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn"] Jul 31 23:02:28.635850 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.635802 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn"] Jul 31 23:02:28.635981 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.635933 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.639368 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.639350 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 23:02:28.639491 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.639354 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 23:02:28.640994 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.640975 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mmk8v\"" Jul 31 23:02:28.650866 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.650812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdrh\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-kube-api-access-6wdrh\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.650963 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.650897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.663410 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.663384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.663487 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.663432 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdrh\" (UniqueName: \"kubernetes.io/projected/613ce901-8efc-41eb-84bf-29effcae867f-kube-api-access-6wdrh\") pod \"cert-manager-webhook-6486ffd997-vjtk4\" (UID: \"613ce901-8efc-41eb-84bf-29effcae867f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.752143 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.752093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.752143 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.752125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.752262 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.752166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxf7\" (UniqueName: \"kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.774126 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.774106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:28.854419 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.853538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxf7\" (UniqueName: \"kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.854419 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.853608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.854419 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.853648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.854419 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.854108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.854419 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.854378 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.869074 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.869048 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxf7\" (UniqueName: \"kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.939753 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.939728 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vjtk4"] Jul 31 23:02:28.942084 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:28.942057 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod613ce901_8efc_41eb_84bf_29effcae867f.slice/crio-06e2027077089de94dd2cac247f19beddbce7b46e3da3274dc8135139b717345 WatchSource:0}: Error finding container 06e2027077089de94dd2cac247f19beddbce7b46e3da3274dc8135139b717345: Status 404 returned error can't find the container with id 06e2027077089de94dd2cac247f19beddbce7b46e3da3274dc8135139b717345 Jul 31 23:02:28.945432 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.945414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:28.990961 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:28.990935 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" event={"ID":"613ce901-8efc-41eb-84bf-29effcae867f","Type":"ContainerStarted","Data":"06e2027077089de94dd2cac247f19beddbce7b46e3da3274dc8135139b717345"} Jul 31 23:02:29.099630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:29.099608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn"] Jul 31 23:02:29.101618 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:29.101593 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368f6955_a212_4135_a3d9_12e51268051a.slice/crio-a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e WatchSource:0}: Error finding container a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e: Status 404 returned error can't find the container with id a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e Jul 31 23:02:30.002654 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:30.002607 2570 generic.go:356] "Generic (PLEG): container finished" podID="368f6955-a212-4135-a3d9-12e51268051a" containerID="aed1c4b608c83581ffac287fb66eb280d4c7e7fa6d615bab801013a271a16043" exitCode=0 Jul 31 23:02:30.003161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:30.002701 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" event={"ID":"368f6955-a212-4135-a3d9-12e51268051a","Type":"ContainerDied","Data":"aed1c4b608c83581ffac287fb66eb280d4c7e7fa6d615bab801013a271a16043"} Jul 31 23:02:30.003161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:30.002800 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" event={"ID":"368f6955-a212-4135-a3d9-12e51268051a","Type":"ContainerStarted","Data":"a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e"} Jul 31 23:02:33.020223 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:33.020185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" event={"ID":"613ce901-8efc-41eb-84bf-29effcae867f","Type":"ContainerStarted","Data":"0fa286a4bb8b8600f70fd6a5efd8f6e24fa60df0dd78179fe7be28cea588c599"} Jul 31 23:02:33.020659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:33.020320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:33.022031 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:33.022001 2570 generic.go:356] "Generic (PLEG): container finished" podID="368f6955-a212-4135-a3d9-12e51268051a" containerID="be0f2bd046d822b0ab912569cb9fefb6bd6d158518c62033261bd96b79698caa" exitCode=0 Jul 31 23:02:33.022158 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:33.022080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" event={"ID":"368f6955-a212-4135-a3d9-12e51268051a","Type":"ContainerDied","Data":"be0f2bd046d822b0ab912569cb9fefb6bd6d158518c62033261bd96b79698caa"} Jul 31 23:02:33.035964 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:33.035918 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" podStartSLOduration=1.676808738 podStartE2EDuration="5.03590019s" totalImagesPullingTime="3.359091452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:02:33.035132215 +0000 UTC m=+340.957399988" watchObservedRunningTime="2026-07-31 23:02:33.03590019 +0000 UTC m=+340.958167963" Jul 31 23:02:34.028355 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:34.028321 2570 generic.go:356] "Generic (PLEG): container finished" podID="368f6955-a212-4135-a3d9-12e51268051a" containerID="0945cc6da5cccc3b473ee75d6b3d720718ee7eaedbaf684d9c131ad669d6825d" exitCode=0 Jul 31 23:02:34.028723 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:34.028399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" event={"ID":"368f6955-a212-4135-a3d9-12e51268051a","Type":"ContainerDied","Data":"0945cc6da5cccc3b473ee75d6b3d720718ee7eaedbaf684d9c131ad669d6825d"} Jul 31 23:02:35.154739 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.154714 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:35.303359 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.303291 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle\") pod \"368f6955-a212-4135-a3d9-12e51268051a\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " Jul 31 23:02:35.303359 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.303337 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util\") pod \"368f6955-a212-4135-a3d9-12e51268051a\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " Jul 31 23:02:35.303506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.303373 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkxf7\" (UniqueName: \"kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7\") pod \"368f6955-a212-4135-a3d9-12e51268051a\" (UID: \"368f6955-a212-4135-a3d9-12e51268051a\") " Jul 31 23:02:35.303668 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.303644 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle" (OuterVolumeSpecName: "bundle") pod "368f6955-a212-4135-a3d9-12e51268051a" (UID: "368f6955-a212-4135-a3d9-12e51268051a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:35.305401 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.305377 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7" (OuterVolumeSpecName: "kube-api-access-kkxf7") pod "368f6955-a212-4135-a3d9-12e51268051a" (UID: "368f6955-a212-4135-a3d9-12e51268051a"). InnerVolumeSpecName "kube-api-access-kkxf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:02:35.309668 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.309640 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util" (OuterVolumeSpecName: "util") pod "368f6955-a212-4135-a3d9-12e51268051a" (UID: "368f6955-a212-4135-a3d9-12e51268051a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:35.404020 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.403995 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:35.404151 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.404016 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/368f6955-a212-4135-a3d9-12e51268051a-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:35.404151 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:35.404045 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkxf7\" (UniqueName: \"kubernetes.io/projected/368f6955-a212-4135-a3d9-12e51268051a-kube-api-access-kkxf7\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:36.039136 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:36.039108 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" Jul 31 23:02:36.039304 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:36.039110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f22ztn" event={"ID":"368f6955-a212-4135-a3d9-12e51268051a","Type":"ContainerDied","Data":"a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e"} Jul 31 23:02:36.039304 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:36.039227 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63882f1527b39ea6fff62fea9bfa514f134d9abd93f8ff38ebf33b06544945e" Jul 31 23:02:39.030530 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:39.030499 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vjtk4" Jul 31 23:02:41.408083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408047 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhg5z"] Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408352 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="pull" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408364 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="pull" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408374 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="extract" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408380 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="extract" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408391 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="util" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408396 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="util" Jul 31 23:02:41.408456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.408441 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="368f6955-a212-4135-a3d9-12e51268051a" containerName="extract" Jul 31 23:02:41.410458 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.410442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.413078 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.413050 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6wqsf\"" Jul 31 23:02:41.420122 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.420101 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhg5z"] Jul 31 23:02:41.548092 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.548057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2np\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-kube-api-access-4b2np\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.548092 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.548099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.648864 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.648809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.649015 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.648929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2np\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-kube-api-access-4b2np\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.657798 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.657768 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.657924 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.657773 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2np\" (UniqueName: \"kubernetes.io/projected/7f46222a-b913-4470-951a-86baaefdc1a4-kube-api-access-4b2np\") pod \"cert-manager-55b4d954d5-jhg5z\" (UID: \"7f46222a-b913-4470-951a-86baaefdc1a4\") " pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.718983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.718951 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jhg5z" Jul 31 23:02:41.850384 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:41.850359 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhg5z"] Jul 31 23:02:41.852738 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:41.852705 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f46222a_b913_4470_951a_86baaefdc1a4.slice/crio-3dbd6cedf5bb3c279d57dba6d895e1aa79d8de7e42568bfba55718b24e69d8e2 WatchSource:0}: Error finding container 3dbd6cedf5bb3c279d57dba6d895e1aa79d8de7e42568bfba55718b24e69d8e2: Status 404 returned error can't find the container with id 3dbd6cedf5bb3c279d57dba6d895e1aa79d8de7e42568bfba55718b24e69d8e2 Jul 31 23:02:42.063450 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:42.063346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jhg5z" event={"ID":"7f46222a-b913-4470-951a-86baaefdc1a4","Type":"ContainerStarted","Data":"e3e2943ea7f7afddced39acd320f2984ccd37deb74813805eba4e5c17ea0443b"} Jul 31 23:02:42.063450 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:42.063402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jhg5z" event={"ID":"7f46222a-b913-4470-951a-86baaefdc1a4","Type":"ContainerStarted","Data":"3dbd6cedf5bb3c279d57dba6d895e1aa79d8de7e42568bfba55718b24e69d8e2"} Jul 31 23:02:42.081047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:42.080987 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jhg5z" podStartSLOduration=1.08097078 podStartE2EDuration="1.08097078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:02:42.080504155 +0000 UTC m=+350.002771925" watchObservedRunningTime="2026-07-31 23:02:42.08097078 +0000 UTC m=+350.003238550" Jul 31 23:02:48.622302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.622213 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc"] Jul 31 23:02:48.624706 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.624691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.627107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.627083 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 23:02:48.628156 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.628138 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 23:02:48.628252 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.628150 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mmk8v\"" Jul 31 23:02:48.633535 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.633510 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc"] Jul 31 23:02:48.705799 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.705767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.705986 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.705858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42j8\" (UniqueName: \"kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.705986 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.705892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.807004 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.806966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.807205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.807030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.807205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.807122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42j8\" (UniqueName: \"kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.807391 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.807368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.807455 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.807433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.815138 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.815099 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42j8\" (UniqueName: \"kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:48.934179 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:48.934086 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:49.061800 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:49.061773 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc"] Jul 31 23:02:49.063766 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:02:49.063741 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5d15b7_9352_4a84_8677_5c69260a13db.slice/crio-1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2 WatchSource:0}: Error finding container 1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2: Status 404 returned error can't find the container with id 1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2 Jul 31 23:02:49.093853 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:49.093810 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" event={"ID":"3a5d15b7-9352-4a84-8677-5c69260a13db","Type":"ContainerStarted","Data":"1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2"} Jul 31 23:02:50.099058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:50.098968 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerID="c0760f52dad4234455b522ba8b078839b90c06261ba265ee4265fd54072eb9ed" exitCode=0 Jul 31 23:02:50.099058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:50.099037 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" event={"ID":"3a5d15b7-9352-4a84-8677-5c69260a13db","Type":"ContainerDied","Data":"c0760f52dad4234455b522ba8b078839b90c06261ba265ee4265fd54072eb9ed"} Jul 31 23:02:51.105436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:51.105393 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerID="a66b27b64bbe422c427f35537c66693509bd53f65b7bf9f41f1f20889fb3fa6c" exitCode=0 Jul 31 23:02:51.105967 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:51.105480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" event={"ID":"3a5d15b7-9352-4a84-8677-5c69260a13db","Type":"ContainerDied","Data":"a66b27b64bbe422c427f35537c66693509bd53f65b7bf9f41f1f20889fb3fa6c"} Jul 31 23:02:52.113600 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:52.113551 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerID="ff5bae828c9baa3ad7306372207b64fb25860768518459fa2a06807000db3fe5" exitCode=0 Jul 31 23:02:52.114078 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:52.113640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" event={"ID":"3a5d15b7-9352-4a84-8677-5c69260a13db","Type":"ContainerDied","Data":"ff5bae828c9baa3ad7306372207b64fb25860768518459fa2a06807000db3fe5"} Jul 31 23:02:53.264113 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.264070 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:02:53.340795 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.340756 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g42j8\" (UniqueName: \"kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8\") pod \"3a5d15b7-9352-4a84-8677-5c69260a13db\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " Jul 31 23:02:53.341022 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.340808 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle\") pod \"3a5d15b7-9352-4a84-8677-5c69260a13db\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " Jul 31 23:02:53.341022 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.340857 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util\") pod \"3a5d15b7-9352-4a84-8677-5c69260a13db\" (UID: \"3a5d15b7-9352-4a84-8677-5c69260a13db\") " Jul 31 23:02:53.341628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.341592 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle" (OuterVolumeSpecName: "bundle") pod "3a5d15b7-9352-4a84-8677-5c69260a13db" (UID: "3a5d15b7-9352-4a84-8677-5c69260a13db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:53.343062 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.343022 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8" (OuterVolumeSpecName: "kube-api-access-g42j8") pod "3a5d15b7-9352-4a84-8677-5c69260a13db" (UID: "3a5d15b7-9352-4a84-8677-5c69260a13db"). InnerVolumeSpecName "kube-api-access-g42j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:02:53.347026 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.346891 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util" (OuterVolumeSpecName: "util") pod "3a5d15b7-9352-4a84-8677-5c69260a13db" (UID: "3a5d15b7-9352-4a84-8677-5c69260a13db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:02:53.442114 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.442076 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:53.442114 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.442107 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5d15b7-9352-4a84-8677-5c69260a13db-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:53.442114 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:53.442116 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g42j8\" (UniqueName: \"kubernetes.io/projected/3a5d15b7-9352-4a84-8677-5c69260a13db-kube-api-access-g42j8\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:02:54.124530 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:54.124492 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" event={"ID":"3a5d15b7-9352-4a84-8677-5c69260a13db","Type":"ContainerDied","Data":"1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2"} Jul 31 23:02:54.124712 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:54.124547 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1907f107f009cc1552f1249dc6bac8ef0eb1c3d36b9b707604456e686f008ad2" Jul 31 23:02:54.124712 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:02:54.124503 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh2hcc" Jul 31 23:03:06.308329 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308288 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq"] Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308574 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="pull" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308590 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="pull" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308611 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="util" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308616 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="util" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308622 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="extract" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308629 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="extract" Jul 31 23:03:06.308780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.308684 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5d15b7-9352-4a84-8677-5c69260a13db" containerName="extract" Jul 31 23:03:06.313484 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.313464 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.316597 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.316568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 23:03:06.316716 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.316606 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 23:03:06.316716 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.316696 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 23:03:06.316998 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.316982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zbcqc\"" Jul 31 23:03:06.317114 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.317097 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 23:03:06.327020 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.327000 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq"] Jul 31 23:03:06.335396 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.335369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.335493 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.335425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vc2p\" (UniqueName: \"kubernetes.io/projected/dc007521-9c05-4f43-99b6-90b02652eb03-kube-api-access-6vc2p\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.335584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.335509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-manifests\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.335584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.335533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-tmp\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.335584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.335553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-webhook-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.436661 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.436635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-manifests\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.436661 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.436664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-tmp\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.436858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.436684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-webhook-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.436858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.436726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.436858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.436750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc2p\" (UniqueName: \"kubernetes.io/projected/dc007521-9c05-4f43-99b6-90b02652eb03-kube-api-access-6vc2p\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.437148 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.437123 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-manifests\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.439036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.439007 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc007521-9c05-4f43-99b6-90b02652eb03-tmp\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.439036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.439034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-webhook-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.439221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.439186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc007521-9c05-4f43-99b6-90b02652eb03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.444404 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.444384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vc2p\" (UniqueName: \"kubernetes.io/projected/dc007521-9c05-4f43-99b6-90b02652eb03-kube-api-access-6vc2p\") pod \"opendatahub-operator-controller-manager-8567dd98f6-v6rxq\" (UID: \"dc007521-9c05-4f43-99b6-90b02652eb03\") " pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.623856 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.623762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:06.762835 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:06.762803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq"] Jul 31 23:03:06.765393 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:06.765370 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc007521_9c05_4f43_99b6_90b02652eb03.slice/crio-a01606dee788c9007b29f1466722e94dfc57f52eb60cb75bc91f9225c2017c9b WatchSource:0}: Error finding container a01606dee788c9007b29f1466722e94dfc57f52eb60cb75bc91f9225c2017c9b: Status 404 returned error can't find the container with id a01606dee788c9007b29f1466722e94dfc57f52eb60cb75bc91f9225c2017c9b Jul 31 23:03:07.186296 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:07.186264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" event={"ID":"dc007521-9c05-4f43-99b6-90b02652eb03","Type":"ContainerStarted","Data":"a01606dee788c9007b29f1466722e94dfc57f52eb60cb75bc91f9225c2017c9b"} Jul 31 23:03:10.202487 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:10.202444 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" event={"ID":"dc007521-9c05-4f43-99b6-90b02652eb03","Type":"ContainerStarted","Data":"9986a53c08f1698a126aca74849c298f1295b439ec77c438e53e47949a130abb"} Jul 31 23:03:11.208429 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:11.208391 2570 generic.go:356] "Generic (PLEG): container finished" podID="dc007521-9c05-4f43-99b6-90b02652eb03" containerID="9986a53c08f1698a126aca74849c298f1295b439ec77c438e53e47949a130abb" exitCode=0 Jul 31 23:03:11.208787 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:11.208477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" event={"ID":"dc007521-9c05-4f43-99b6-90b02652eb03","Type":"ContainerDied","Data":"9986a53c08f1698a126aca74849c298f1295b439ec77c438e53e47949a130abb"} Jul 31 23:03:12.214894 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:12.214854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" event={"ID":"dc007521-9c05-4f43-99b6-90b02652eb03","Type":"ContainerStarted","Data":"dcf98f3d8ebe17db268cc8400c9a8e69c359fa3fdfc6d37dae2f61daf838b68e"} Jul 31 23:03:12.215270 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:12.214956 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:12.240203 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:12.240148 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" podStartSLOduration=2.016370528 podStartE2EDuration="6.240128371s" totalImagesPullingTime="3.159770014s" totalInitContainerRuntime="1.063987829s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:12.236935067 +0000 UTC m=+380.159202839" watchObservedRunningTime="2026-07-31 23:03:12.240128371 +0000 UTC m=+380.162396142" Jul 31 23:03:23.220837 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:23.220792 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8567dd98f6-v6rxq" Jul 31 23:03:26.113550 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.113509 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl"] Jul 31 23:03:26.118317 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.118294 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.122368 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.122346 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 23:03:26.122510 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.122401 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 23:03:26.122608 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.122589 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 23:03:26.122855 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.122816 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 23:03:26.124384 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.124366 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vjkqq\"" Jul 31 23:03:26.134061 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.134045 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl"] Jul 31 23:03:26.192527 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.192498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.192641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.192550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db0525c-dadd-4b88-95e5-7392a88126ad-tmp\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.192641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.192601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24d2\" (UniqueName: \"kubernetes.io/projected/2db0525c-dadd-4b88-95e5-7392a88126ad-kube-api-access-j24d2\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.293694 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.293658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.293849 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.293709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db0525c-dadd-4b88-95e5-7392a88126ad-tmp\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.293849 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.293742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24d2\" (UniqueName: \"kubernetes.io/projected/2db0525c-dadd-4b88-95e5-7392a88126ad-kube-api-access-j24d2\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.293849 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:03:26.293839 2570 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 31 23:03:26.293971 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:03:26.293912 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs podName:2db0525c-dadd-4b88-95e5-7392a88126ad nodeName:}" failed. No retries permitted until 2026-07-31 23:03:26.793891889 +0000 UTC m=+394.716159650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs") pod "kube-auth-proxy-85c4c7ff5b-t8lgl" (UID: "2db0525c-dadd-4b88-95e5-7392a88126ad") : secret "kube-auth-proxy-tls" not found Jul 31 23:03:26.295983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.295957 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db0525c-dadd-4b88-95e5-7392a88126ad-tmp\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.310633 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.310607 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24d2\" (UniqueName: \"kubernetes.io/projected/2db0525c-dadd-4b88-95e5-7392a88126ad-kube-api-access-j24d2\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.400777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.400704 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56v8k"] Jul 31 23:03:26.404038 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.404023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.406778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.406756 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 31 23:03:26.406893 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.406758 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k9lkn\"" Jul 31 23:03:26.414763 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.414736 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56v8k"] Jul 31 23:03:26.496117 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.496085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e934743-011d-4538-b7b1-2bb8e70c26e0-cert\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.496291 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.496145 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6t5\" (UniqueName: \"kubernetes.io/projected/8e934743-011d-4538-b7b1-2bb8e70c26e0-kube-api-access-2h6t5\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.596589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.596554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e934743-011d-4538-b7b1-2bb8e70c26e0-cert\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.596729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.596597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6t5\" (UniqueName: \"kubernetes.io/projected/8e934743-011d-4538-b7b1-2bb8e70c26e0-kube-api-access-2h6t5\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.599013 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.598984 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e934743-011d-4538-b7b1-2bb8e70c26e0-cert\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.607767 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.607745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6t5\" (UniqueName: \"kubernetes.io/projected/8e934743-011d-4538-b7b1-2bb8e70c26e0-kube-api-access-2h6t5\") pod \"odh-model-controller-7998c67dc4-56v8k\" (UID: \"8e934743-011d-4538-b7b1-2bb8e70c26e0\") " pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.715921 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.715887 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:26.798893 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.798861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.801213 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.801186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2db0525c-dadd-4b88-95e5-7392a88126ad-tls-certs\") pod \"kube-auth-proxy-85c4c7ff5b-t8lgl\" (UID: \"2db0525c-dadd-4b88-95e5-7392a88126ad\") " pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:26.865046 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:26.865018 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56v8k"] Jul 31 23:03:26.867315 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:26.867288 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e934743_011d_4538_b7b1_2bb8e70c26e0.slice/crio-90e82a5cbaf87f7661b9af745173715ea5b9b88062125fab57323e5f207e69df WatchSource:0}: Error finding container 90e82a5cbaf87f7661b9af745173715ea5b9b88062125fab57323e5f207e69df: Status 404 returned error can't find the container with id 90e82a5cbaf87f7661b9af745173715ea5b9b88062125fab57323e5f207e69df Jul 31 23:03:27.029241 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.029162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" Jul 31 23:03:27.160308 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.160143 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl"] Jul 31 23:03:27.162729 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:27.162693 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db0525c_dadd_4b88_95e5_7392a88126ad.slice/crio-f33aded7e2d9db5b5be6945cb162151e7a8b54e74672998f2201d9584a5482f5 WatchSource:0}: Error finding container f33aded7e2d9db5b5be6945cb162151e7a8b54e74672998f2201d9584a5482f5: Status 404 returned error can't find the container with id f33aded7e2d9db5b5be6945cb162151e7a8b54e74672998f2201d9584a5482f5 Jul 31 23:03:27.280008 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.279922 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" event={"ID":"2db0525c-dadd-4b88-95e5-7392a88126ad","Type":"ContainerStarted","Data":"f33aded7e2d9db5b5be6945cb162151e7a8b54e74672998f2201d9584a5482f5"} Jul 31 23:03:27.281343 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.281306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" event={"ID":"8e934743-011d-4538-b7b1-2bb8e70c26e0","Type":"ContainerStarted","Data":"90e82a5cbaf87f7661b9af745173715ea5b9b88062125fab57323e5f207e69df"} Jul 31 23:03:27.368454 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.368420 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:27.373477 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.373458 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.376154 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.376131 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 31 23:03:27.376294 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.376133 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 31 23:03:27.376294 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.376131 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 23:03:27.376409 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.376372 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pzrd6\"" Jul 31 23:03:27.376486 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.376459 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 31 23:03:27.386817 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.386791 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:27.504079 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxqt\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504300 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504300 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504290 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504367 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.504399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.504388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.605209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.605209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.605547 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.605607 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.605607 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.606059 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.606059 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.605904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxqt\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.606322 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.606298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.608938 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.608580 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.608938 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.608901 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.613053 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.612265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.613053 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.612504 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.614734 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.614709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.614852 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.614800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxqt\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt\") pod \"istiod-openshift-gateway-7b7995d5cd-cs72c\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.683761 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.683724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:27.866267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:27.866187 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:27.870482 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:27.870336 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a1b8aa_98b9_4b3a_ac93_9e563dc8f3a0.slice/crio-67e4ec061d0b8c2af5edb203d27943067865e4191561f320d8d5e75d5a179e59 WatchSource:0}: Error finding container 67e4ec061d0b8c2af5edb203d27943067865e4191561f320d8d5e75d5a179e59: Status 404 returned error can't find the container with id 67e4ec061d0b8c2af5edb203d27943067865e4191561f320d8d5e75d5a179e59 Jul 31 23:03:28.292938 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:28.292855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" event={"ID":"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0","Type":"ContainerStarted","Data":"67e4ec061d0b8c2af5edb203d27943067865e4191561f320d8d5e75d5a179e59"} Jul 31 23:03:32.918683 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:32.918650 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:03:32.919072 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:32.918713 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:03:33.321172 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.321130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" event={"ID":"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0","Type":"ContainerStarted","Data":"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1"} Jul 31 23:03:33.321364 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.321308 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:33.323172 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.323142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" event={"ID":"2db0525c-dadd-4b88-95e5-7392a88126ad","Type":"ContainerStarted","Data":"d39522e89a11cb8d6d3b4e7c7d90d8f6708dfa7dfcd798afb70983a17f9e0177"} Jul 31 23:03:33.323508 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.323484 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cs72c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 23:03:33.323629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.323543 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 23:03:33.324761 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.324739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" event={"ID":"8e934743-011d-4538-b7b1-2bb8e70c26e0","Type":"ContainerStarted","Data":"a28e93c304233454f07fb8ea6225018d5124fe94b61ae8c26dc0e95a77629bf4"} Jul 31 23:03:33.324885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.324869 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:33.346762 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.346718 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" podStartSLOduration=1.30565176 podStartE2EDuration="6.34670444s" totalImagesPullingTime="5.04105268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:33.345343476 +0000 UTC m=+401.267611249" watchObservedRunningTime="2026-07-31 23:03:33.34670444 +0000 UTC m=+401.268972210" Jul 31 23:03:33.367307 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.367259 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85c4c7ff5b-t8lgl" podStartSLOduration=1.604031164 podStartE2EDuration="7.367247097s" totalImagesPullingTime="5.763215933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:33.365622335 +0000 UTC m=+401.287890107" watchObservedRunningTime="2026-07-31 23:03:33.367247097 +0000 UTC m=+401.289514901" Jul 31 23:03:33.387865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:33.387507 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" podStartSLOduration=1.3403504019999999 podStartE2EDuration="7.387478557s" totalImagesPullingTime="6.047128155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:33.38732737 +0000 UTC m=+401.309595141" watchObservedRunningTime="2026-07-31 23:03:33.387478557 +0000 UTC m=+401.309746328" Jul 31 23:03:34.330393 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:34.330356 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cs72c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 23:03:34.330763 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:34.330422 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 23:03:36.564247 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.564212 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql"] Jul 31 23:03:36.569112 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.569081 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.573718 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573693 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 31 23:03:36.573718 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573704 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 23:03:36.573903 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573728 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 31 23:03:36.573903 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573731 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 23:03:36.573903 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573778 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 31 23:03:36.574086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.573902 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kwgbf\"" Jul 31 23:03:36.576296 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.576274 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql"] Jul 31 23:03:36.684281 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.684255 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.684417 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.684298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wb56\" (UniqueName: \"kubernetes.io/projected/ec1f1587-f293-49ef-9b3f-593321eebf51-kube-api-access-4wb56\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.684417 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.684395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-metrics-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.684532 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.684465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec1f1587-f293-49ef-9b3f-593321eebf51-manager-config\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.784795 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.784765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec1f1587-f293-49ef-9b3f-593321eebf51-manager-config\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.784978 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.784836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.784978 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.784866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wb56\" (UniqueName: \"kubernetes.io/projected/ec1f1587-f293-49ef-9b3f-593321eebf51-kube-api-access-4wb56\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.784978 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.784911 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-metrics-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.785838 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.785801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec1f1587-f293-49ef-9b3f-593321eebf51-manager-config\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.787592 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.787570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-metrics-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.787703 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.787665 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec1f1587-f293-49ef-9b3f-593321eebf51-cert\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.795522 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.795503 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wb56\" (UniqueName: \"kubernetes.io/projected/ec1f1587-f293-49ef-9b3f-593321eebf51-kube-api-access-4wb56\") pod \"lws-controller-manager-6b79896bff-n26ql\" (UID: \"ec1f1587-f293-49ef-9b3f-593321eebf51\") " pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:36.880721 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:36.880653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:37.012248 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:37.012223 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql"] Jul 31 23:03:37.013811 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:37.013784 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1f1587_f293_49ef_9b3f_593321eebf51.slice/crio-05b7ac8354d271b7d2c6f3e553713dcc52f6840b1673339529e272d57245143f WatchSource:0}: Error finding container 05b7ac8354d271b7d2c6f3e553713dcc52f6840b1673339529e272d57245143f: Status 404 returned error can't find the container with id 05b7ac8354d271b7d2c6f3e553713dcc52f6840b1673339529e272d57245143f Jul 31 23:03:37.331034 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:37.331005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:37.344424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:37.344397 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" event={"ID":"ec1f1587-f293-49ef-9b3f-593321eebf51","Type":"ContainerStarted","Data":"05b7ac8354d271b7d2c6f3e553713dcc52f6840b1673339529e272d57245143f"} Jul 31 23:03:39.354989 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.354951 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" event={"ID":"ec1f1587-f293-49ef-9b3f-593321eebf51","Type":"ContainerStarted","Data":"14446305a1ff9e801dacd6880b9614ac73cdd191438e864346e3e413bab0d8bc"} Jul 31 23:03:39.355364 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.355033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:39.382172 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.382125 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" podStartSLOduration=1.7776385380000002 podStartE2EDuration="3.38211177s" totalImagesPullingTime="1.604473232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:39.381457621 +0000 UTC m=+407.303725393" watchObservedRunningTime="2026-07-31 23:03:39.38211177 +0000 UTC m=+407.304379540" Jul 31 23:03:39.681420 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.681348 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:39.681563 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.681517 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" containerID="cri-o://0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1" gracePeriod=30 Jul 31 23:03:39.941099 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:39.941075 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:40.015192 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015162 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015316 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015316 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015237 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015316 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015257 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015316 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015280 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjxqt\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015316 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015307 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015561 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015349 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs\") pod \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\" (UID: \"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0\") " Jul 31 23:03:40.015692 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.015653 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:03:40.017623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.017579 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs" (OuterVolumeSpecName: "local-certs") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:03:40.017623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.017594 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts" (OuterVolumeSpecName: "cacerts") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:03:40.017798 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.017706 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:03:40.017798 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.017707 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:03:40.017936 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.017899 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:03:40.018024 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.018000 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt" (OuterVolumeSpecName: "kube-api-access-kjxqt") pod "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" (UID: "54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0"). InnerVolumeSpecName "kube-api-access-kjxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:03:40.116495 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116475 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-cacerts\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116495 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116495 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-local-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116505 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-token\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116514 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-dns-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116522 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-kubeconfig\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116531 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-istio-csr-ca-configmap\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.116635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.116539 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxqt\" (UniqueName: \"kubernetes.io/projected/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0-kube-api-access-kjxqt\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:03:40.362676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.362393 2570 generic.go:356] "Generic (PLEG): container finished" podID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerID="0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1" exitCode=0 Jul 31 23:03:40.362676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.362467 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" Jul 31 23:03:40.362676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.362484 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" event={"ID":"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0","Type":"ContainerDied","Data":"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1"} Jul 31 23:03:40.362676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.362555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c" event={"ID":"54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0","Type":"ContainerDied","Data":"67e4ec061d0b8c2af5edb203d27943067865e4191561f320d8d5e75d5a179e59"} Jul 31 23:03:40.362676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.362571 2570 scope.go:117] "RemoveContainer" containerID="0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1" Jul 31 23:03:40.377658 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.377465 2570 scope.go:117] "RemoveContainer" containerID="0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1" Jul 31 23:03:40.378036 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:03:40.377955 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1\": container with ID starting with 0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1 not found: ID does not exist" containerID="0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1" Jul 31 23:03:40.378036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.377997 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1"} err="failed to get container status \"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1\": rpc error: code = NotFound desc = could not find container \"0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1\": container with ID starting with 0e258ea8dd9ef4305bdac88c2a28ffedadcb1aa7cb674a1ed36bea011d89d6f1 not found: ID does not exist" Jul 31 23:03:40.398249 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.398199 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:40.400885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.400863 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cs72c"] Jul 31 23:03:40.537864 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:40.537834 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" path="/var/lib/kubelet/pods/54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0/volumes" Jul 31 23:03:44.332073 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:44.332040 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-56v8k" Jul 31 23:03:50.366021 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:50.365989 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6b79896bff-n26ql" Jul 31 23:03:54.171903 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.171857 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84494b959b-mllr9"] Jul 31 23:03:54.172281 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.172201 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" Jul 31 23:03:54.172281 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.172215 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" Jul 31 23:03:54.172281 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.172280 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="54a1b8aa-98b9-4b3a-ac93-9e563dc8f3a0" containerName="discovery" Jul 31 23:03:54.175763 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.175734 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.185926 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.185902 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84494b959b-mllr9"] Jul 31 23:03:54.327764 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-service-ca\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.327916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-oauth-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.327916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-trusted-ca-bundle\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.327916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-oauth-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.328035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4q2n\" (UniqueName: \"kubernetes.io/projected/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-kube-api-access-f4q2n\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.328035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.327984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.328035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.328000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429044 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.428986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-service-ca\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429044 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-oauth-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-trusted-ca-bundle\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-oauth-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429113 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4q2n\" (UniqueName: \"kubernetes.io/projected/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-kube-api-access-f4q2n\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.429912 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429880 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-oauth-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.430032 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-trusted-ca-bundle\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.430032 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-service-ca\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.430032 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.429966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.431498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.431480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-oauth-config\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.431714 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.431693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-console-serving-cert\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.437374 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.437351 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4q2n\" (UniqueName: \"kubernetes.io/projected/e29304de-3d07-4a1c-8ea9-c3d73f1bf551-kube-api-access-f4q2n\") pod \"console-84494b959b-mllr9\" (UID: \"e29304de-3d07-4a1c-8ea9-c3d73f1bf551\") " pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.486311 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.486289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:03:54.617923 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:54.617805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84494b959b-mllr9"] Jul 31 23:03:54.620005 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:03:54.619977 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29304de_3d07_4a1c_8ea9_c3d73f1bf551.slice/crio-d8ae46f9a825155750c1b0c7adf01b4bd4f514f70cbe9bfb4f9f63ef79cfaa0e WatchSource:0}: Error finding container d8ae46f9a825155750c1b0c7adf01b4bd4f514f70cbe9bfb4f9f63ef79cfaa0e: Status 404 returned error can't find the container with id d8ae46f9a825155750c1b0c7adf01b4bd4f514f70cbe9bfb4f9f63ef79cfaa0e Jul 31 23:03:55.429328 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:55.429293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494b959b-mllr9" event={"ID":"e29304de-3d07-4a1c-8ea9-c3d73f1bf551","Type":"ContainerStarted","Data":"0c8c53f91370cda0ee8dbec78820649924ddf1bb56a311d7e2035bd603c1cb60"} Jul 31 23:03:55.429732 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:55.429342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494b959b-mllr9" event={"ID":"e29304de-3d07-4a1c-8ea9-c3d73f1bf551","Type":"ContainerStarted","Data":"d8ae46f9a825155750c1b0c7adf01b4bd4f514f70cbe9bfb4f9f63ef79cfaa0e"} Jul 31 23:03:55.446441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:03:55.446386 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84494b959b-mllr9" podStartSLOduration=1.446367226 podStartE2EDuration="1.446367226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:03:55.446108532 +0000 UTC m=+423.368376305" watchObservedRunningTime="2026-07-31 23:03:55.446367226 +0000 UTC m=+423.368634998" Jul 31 23:04:04.487287 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:04.487250 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:04:04.487738 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:04.487519 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:04:04.492057 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:04.492033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:04:05.476414 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:05.476388 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84494b959b-mllr9" Jul 31 23:04:05.545859 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:05.545809 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:04:29.998901 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:29.998873 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk"] Jul 31 23:04:30.001381 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.001355 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.003951 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.003931 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 23:04:30.005035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.005016 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 23:04:30.005136 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.005031 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mmk8v\"" Jul 31 23:04:30.009650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.009629 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk"] Jul 31 23:04:30.081994 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.081960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.082099 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.081997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sc4g\" (UniqueName: \"kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.082099 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.082021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.098211 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.098181 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg"] Jul 31 23:04:30.100568 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.100551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.109741 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.109721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg"] Jul 31 23:04:30.182350 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc4g\" (UniqueName: \"kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.182436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k6s\" (UniqueName: \"kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.182436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.182436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.182588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.182588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182568 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.182717 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182704 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.182794 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.182777 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.193427 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.193401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc4g\" (UniqueName: \"kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.198953 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.198931 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68"] Jul 31 23:04:30.201940 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.201922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.209017 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.208996 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68"] Jul 31 23:04:30.283586 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfhc\" (UniqueName: \"kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.283586 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.283722 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.283722 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.283722 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.283863 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k6s\" (UniqueName: \"kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.283913 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.283997 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.283982 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.295313 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.295290 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k6s\" (UniqueName: \"kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.301533 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.301505 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq"] Jul 31 23:04:30.304195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.304180 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.310432 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.310403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:30.311859 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.311836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq"] Jul 31 23:04:30.385046 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.384910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.385046 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.384962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.385046 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.384999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.385046 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.385035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlz5\" (UniqueName: \"kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.385367 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.385072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfhc\" (UniqueName: \"kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.385367 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.385108 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.385367 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.385362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.385515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.385403 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.404093 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.404033 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfhc\" (UniqueName: \"kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.410623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.410335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:30.441133 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.441104 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk"] Jul 31 23:04:30.444456 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:04:30.444430 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5bdf9a_ea47_445d_a67f_826681d9a70e.slice/crio-c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c WatchSource:0}: Error finding container c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c: Status 404 returned error can't find the container with id c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c Jul 31 23:04:30.486004 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.485975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.486122 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.486020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlz5\" (UniqueName: \"kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.486122 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.486071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.486400 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.486373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.486482 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.486401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.498561 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.498533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlz5\" (UniqueName: \"kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.511881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.511754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:30.545047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.544959 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg"] Jul 31 23:04:30.548271 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:04:30.548244 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5306e963_6dfa_4d95_82e1_4895a08ac1ba.slice/crio-4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14 WatchSource:0}: Error finding container 4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14: Status 404 returned error can't find the container with id 4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14 Jul 31 23:04:30.568500 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.568456 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6649ff6db9-7xf5q" podUID="1eba0df4-71d2-4eee-a837-83157481d4ab" containerName="console" containerID="cri-o://b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e" gracePeriod=15 Jul 31 23:04:30.580865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.580833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" event={"ID":"5306e963-6dfa-4d95-82e1-4895a08ac1ba","Type":"ContainerStarted","Data":"4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14"} Jul 31 23:04:30.582442 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.582417 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerID="216ecda43791c716b679fa84eca2ce260258a9b7fb53a7d687df3807e00024e5" exitCode=0 Jul 31 23:04:30.582550 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.582470 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" event={"ID":"ab5bdf9a-ea47-445d-a67f-826681d9a70e","Type":"ContainerDied","Data":"216ecda43791c716b679fa84eca2ce260258a9b7fb53a7d687df3807e00024e5"} Jul 31 23:04:30.582550 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.582508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" event={"ID":"ab5bdf9a-ea47-445d-a67f-826681d9a70e","Type":"ContainerStarted","Data":"c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c"} Jul 31 23:04:30.613086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.613062 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:30.655056 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.655026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68"] Jul 31 23:04:30.656037 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:04:30.656002 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837c5c90_9418_42d4_821e_941b0e3658f9.slice/crio-94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f WatchSource:0}: Error finding container 94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f: Status 404 returned error can't find the container with id 94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f Jul 31 23:04:30.762047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.761921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq"] Jul 31 23:04:30.794690 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:04:30.794662 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e176e0_e101_48c5_bf58_849acc813fe0.slice/crio-610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2 WatchSource:0}: Error finding container 610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2: Status 404 returned error can't find the container with id 610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2 Jul 31 23:04:30.873146 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.873124 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:04:30.889515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889494 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889620 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889532 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889620 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889568 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5m8c\" (UniqueName: \"kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889620 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889624 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889660 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.889778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889693 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config\") pod \"1eba0df4-71d2-4eee-a837-83157481d4ab\" (UID: \"1eba0df4-71d2-4eee-a837-83157481d4ab\") " Jul 31 23:04:30.890007 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.889958 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:04:30.890142 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.890121 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:04:30.890199 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.890146 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config" (OuterVolumeSpecName: "console-config") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:04:30.890199 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.890177 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:04:30.892141 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.892096 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:04:30.892141 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.892114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:04:30.892141 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.892130 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c" (OuterVolumeSpecName: "kube-api-access-c5m8c") pod "1eba0df4-71d2-4eee-a837-83157481d4ab" (UID: "1eba0df4-71d2-4eee-a837-83157481d4ab"). InnerVolumeSpecName "kube-api-access-c5m8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:04:30.990474 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990451 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990476 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-console-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990490 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5m8c\" (UniqueName: \"kubernetes.io/projected/1eba0df4-71d2-4eee-a837-83157481d4ab-kube-api-access-c5m8c\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990503 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990515 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-oauth-serving-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990527 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eba0df4-71d2-4eee-a837-83157481d4ab-trusted-ca-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:30.990580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:30.990540 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eba0df4-71d2-4eee-a837-83157481d4ab-console-oauth-config\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:31.588454 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.588391 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerID="5c55e1ad31074a54540ebe62b807e7eabc3d2fbecda848df07a7230d76ce33f8" exitCode=0 Jul 31 23:04:31.588790 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.588470 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" event={"ID":"d9e176e0-e101-48c5-bf58-849acc813fe0","Type":"ContainerDied","Data":"5c55e1ad31074a54540ebe62b807e7eabc3d2fbecda848df07a7230d76ce33f8"} Jul 31 23:04:31.588790 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.588519 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" event={"ID":"d9e176e0-e101-48c5-bf58-849acc813fe0","Type":"ContainerStarted","Data":"610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2"} Jul 31 23:04:31.590195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.590167 2570 generic.go:356] "Generic (PLEG): container finished" podID="837c5c90-9418-42d4-821e-941b0e3658f9" containerID="421c0f5c03253aed39433b1053a8c34b52d180e5f2968d63c338adaf4839ff36" exitCode=0 Jul 31 23:04:31.590327 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.590236 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" event={"ID":"837c5c90-9418-42d4-821e-941b0e3658f9","Type":"ContainerDied","Data":"421c0f5c03253aed39433b1053a8c34b52d180e5f2968d63c338adaf4839ff36"} Jul 31 23:04:31.590327 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.590274 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" event={"ID":"837c5c90-9418-42d4-821e-941b0e3658f9","Type":"ContainerStarted","Data":"94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f"} Jul 31 23:04:31.591937 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.591892 2570 generic.go:356] "Generic (PLEG): container finished" podID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerID="3a396d841a1278e0bdb34c17c6f9fafa0c987f4d23aff9686fd5849cac95c745" exitCode=0 Jul 31 23:04:31.592038 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.591975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" event={"ID":"5306e963-6dfa-4d95-82e1-4895a08ac1ba","Type":"ContainerDied","Data":"3a396d841a1278e0bdb34c17c6f9fafa0c987f4d23aff9686fd5849cac95c745"} Jul 31 23:04:31.593503 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.593427 2570 generic.go:356] "Generic (PLEG): container finished" podID="1eba0df4-71d2-4eee-a837-83157481d4ab" containerID="b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e" exitCode=0 Jul 31 23:04:31.593503 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.593459 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6649ff6db9-7xf5q" Jul 31 23:04:31.593503 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.593486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649ff6db9-7xf5q" event={"ID":"1eba0df4-71d2-4eee-a837-83157481d4ab","Type":"ContainerDied","Data":"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e"} Jul 31 23:04:31.593719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.593539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6649ff6db9-7xf5q" event={"ID":"1eba0df4-71d2-4eee-a837-83157481d4ab","Type":"ContainerDied","Data":"cbe71356a9f33155e67b76f9adc48a2ab69637658f8da6448add864db57cb946"} Jul 31 23:04:31.593719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.593569 2570 scope.go:117] "RemoveContainer" containerID="b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e" Jul 31 23:04:31.596073 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.595863 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerID="0307f088c9333be869d5c6dea22075cdf064722c2e9b248f1e90c1daeceef6d0" exitCode=0 Jul 31 23:04:31.596073 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.595895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" event={"ID":"ab5bdf9a-ea47-445d-a67f-826681d9a70e","Type":"ContainerDied","Data":"0307f088c9333be869d5c6dea22075cdf064722c2e9b248f1e90c1daeceef6d0"} Jul 31 23:04:31.603221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.603202 2570 scope.go:117] "RemoveContainer" containerID="b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e" Jul 31 23:04:31.603477 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:04:31.603449 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e\": container with ID starting with b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e not found: ID does not exist" containerID="b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e" Jul 31 23:04:31.603571 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.603484 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e"} err="failed to get container status \"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e\": rpc error: code = NotFound desc = could not find container \"b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e\": container with ID starting with b79b2a1c486c4f9068d6a53666e16ff9dc4a33054974d357ac1918883952f61e not found: ID does not exist" Jul 31 23:04:31.654362 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.654339 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:04:31.658880 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:31.658859 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6649ff6db9-7xf5q"] Jul 31 23:04:32.539874 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.539208 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eba0df4-71d2-4eee-a837-83157481d4ab" path="/var/lib/kubelet/pods/1eba0df4-71d2-4eee-a837-83157481d4ab/volumes" Jul 31 23:04:32.603075 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.603015 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerID="11b6b4f08688c024cd108d5d86d8262f43eedee9c52a05ac0ce6756e643a6333" exitCode=0 Jul 31 23:04:32.603372 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.603096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" event={"ID":"ab5bdf9a-ea47-445d-a67f-826681d9a70e","Type":"ContainerDied","Data":"11b6b4f08688c024cd108d5d86d8262f43eedee9c52a05ac0ce6756e643a6333"} Jul 31 23:04:32.604639 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.604614 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerID="1cb9664e63801eccf78229600d6936ea50c821c29a84b42ae41ce81e872f090d" exitCode=0 Jul 31 23:04:32.604731 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.604642 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" event={"ID":"d9e176e0-e101-48c5-bf58-849acc813fe0","Type":"ContainerDied","Data":"1cb9664e63801eccf78229600d6936ea50c821c29a84b42ae41ce81e872f090d"} Jul 31 23:04:32.606308 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.606288 2570 generic.go:356] "Generic (PLEG): container finished" podID="837c5c90-9418-42d4-821e-941b0e3658f9" containerID="80588d0554ed1c0a4eb5f1ac4a4c515c17dd59ae4e32691a85f7db7763d7f8d0" exitCode=0 Jul 31 23:04:32.606388 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.606366 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" event={"ID":"837c5c90-9418-42d4-821e-941b0e3658f9","Type":"ContainerDied","Data":"80588d0554ed1c0a4eb5f1ac4a4c515c17dd59ae4e32691a85f7db7763d7f8d0"} Jul 31 23:04:32.608157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.608136 2570 generic.go:356] "Generic (PLEG): container finished" podID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerID="a98bc8232b8967ccd0a42577a2dcf4e5bd2ab3e5aca63bdd4ebd8a42ae940025" exitCode=0 Jul 31 23:04:32.608244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:32.608171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" event={"ID":"5306e963-6dfa-4d95-82e1-4895a08ac1ba","Type":"ContainerDied","Data":"a98bc8232b8967ccd0a42577a2dcf4e5bd2ab3e5aca63bdd4ebd8a42ae940025"} Jul 31 23:04:33.614765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.614728 2570 generic.go:356] "Generic (PLEG): container finished" podID="837c5c90-9418-42d4-821e-941b0e3658f9" containerID="e70df28447987d9b6aef2da69837a916bce4bb41ba734e6c985f9eec82168743" exitCode=0 Jul 31 23:04:33.615159 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.614817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" event={"ID":"837c5c90-9418-42d4-821e-941b0e3658f9","Type":"ContainerDied","Data":"e70df28447987d9b6aef2da69837a916bce4bb41ba734e6c985f9eec82168743"} Jul 31 23:04:33.616685 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.616664 2570 generic.go:356] "Generic (PLEG): container finished" podID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerID="e7d9ad1c3e8c550fd28adc7ed4bd243856a9584e5661ef863caff0a97af8062b" exitCode=0 Jul 31 23:04:33.616794 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.616741 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" event={"ID":"5306e963-6dfa-4d95-82e1-4895a08ac1ba","Type":"ContainerDied","Data":"e7d9ad1c3e8c550fd28adc7ed4bd243856a9584e5661ef863caff0a97af8062b"} Jul 31 23:04:33.618839 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.618802 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerID="06f8ecb99e1b6c230a6566f8a0413ef4fb43abd406ee8b52248b704ebe71d0bb" exitCode=0 Jul 31 23:04:33.618928 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.618874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" event={"ID":"d9e176e0-e101-48c5-bf58-849acc813fe0","Type":"ContainerDied","Data":"06f8ecb99e1b6c230a6566f8a0413ef4fb43abd406ee8b52248b704ebe71d0bb"} Jul 31 23:04:33.741940 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.741918 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:33.816886 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.816860 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle\") pod \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " Jul 31 23:04:33.816990 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.816910 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sc4g\" (UniqueName: \"kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g\") pod \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " Jul 31 23:04:33.816990 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.816949 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util\") pod \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\" (UID: \"ab5bdf9a-ea47-445d-a67f-826681d9a70e\") " Jul 31 23:04:33.817484 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.817455 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle" (OuterVolumeSpecName: "bundle") pod "ab5bdf9a-ea47-445d-a67f-826681d9a70e" (UID: "ab5bdf9a-ea47-445d-a67f-826681d9a70e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:33.818885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.818865 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g" (OuterVolumeSpecName: "kube-api-access-6sc4g") pod "ab5bdf9a-ea47-445d-a67f-826681d9a70e" (UID: "ab5bdf9a-ea47-445d-a67f-826681d9a70e"). InnerVolumeSpecName "kube-api-access-6sc4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:04:33.825648 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.825626 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util" (OuterVolumeSpecName: "util") pod "ab5bdf9a-ea47-445d-a67f-826681d9a70e" (UID: "ab5bdf9a-ea47-445d-a67f-826681d9a70e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:33.917808 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.917754 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:33.917808 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.917777 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sc4g\" (UniqueName: \"kubernetes.io/projected/ab5bdf9a-ea47-445d-a67f-826681d9a70e-kube-api-access-6sc4g\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:33.917808 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:33.917789 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab5bdf9a-ea47-445d-a67f-826681d9a70e-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:34.630956 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.630927 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" Jul 31 23:04:34.631260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.630979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870538dbxk" event={"ID":"ab5bdf9a-ea47-445d-a67f-826681d9a70e","Type":"ContainerDied","Data":"c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c"} Jul 31 23:04:34.631260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.631039 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43f8f4254dc2aa6e661bfb668e0e935d6b15ad86c4b4c223e925323485c980c" Jul 31 23:04:34.759318 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.759295 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:34.806349 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.806301 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:34.809494 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.809477 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:34.825727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.825705 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle\") pod \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " Jul 31 23:04:34.825857 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.825764 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util\") pod \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " Jul 31 23:04:34.825857 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.825803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6k6s\" (UniqueName: \"kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s\") pod \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\" (UID: \"5306e963-6dfa-4d95-82e1-4895a08ac1ba\") " Jul 31 23:04:34.826328 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.826298 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle" (OuterVolumeSpecName: "bundle") pod "5306e963-6dfa-4d95-82e1-4895a08ac1ba" (UID: "5306e963-6dfa-4d95-82e1-4895a08ac1ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:34.828009 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.827981 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s" (OuterVolumeSpecName: "kube-api-access-q6k6s") pod "5306e963-6dfa-4d95-82e1-4895a08ac1ba" (UID: "5306e963-6dfa-4d95-82e1-4895a08ac1ba"). InnerVolumeSpecName "kube-api-access-q6k6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:04:34.831592 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.831567 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util" (OuterVolumeSpecName: "util") pod "5306e963-6dfa-4d95-82e1-4895a08ac1ba" (UID: "5306e963-6dfa-4d95-82e1-4895a08ac1ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:34.927207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927186 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util\") pod \"d9e176e0-e101-48c5-bf58-849acc813fe0\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " Jul 31 23:04:34.927309 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927219 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dlz5\" (UniqueName: \"kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5\") pod \"d9e176e0-e101-48c5-bf58-849acc813fe0\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " Jul 31 23:04:34.927309 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927238 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle\") pod \"837c5c90-9418-42d4-821e-941b0e3658f9\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " Jul 31 23:04:34.927309 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927278 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util\") pod \"837c5c90-9418-42d4-821e-941b0e3658f9\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " Jul 31 23:04:34.927309 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927303 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cfhc\" (UniqueName: \"kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc\") pod \"837c5c90-9418-42d4-821e-941b0e3658f9\" (UID: \"837c5c90-9418-42d4-821e-941b0e3658f9\") " Jul 31 23:04:34.927519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927319 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle\") pod \"d9e176e0-e101-48c5-bf58-849acc813fe0\" (UID: \"d9e176e0-e101-48c5-bf58-849acc813fe0\") " Jul 31 23:04:34.927519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927466 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:34.927519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927482 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6k6s\" (UniqueName: \"kubernetes.io/projected/5306e963-6dfa-4d95-82e1-4895a08ac1ba-kube-api-access-q6k6s\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:34.927519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927496 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5306e963-6dfa-4d95-82e1-4895a08ac1ba-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:34.927879 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927848 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle" (OuterVolumeSpecName: "bundle") pod "837c5c90-9418-42d4-821e-941b0e3658f9" (UID: "837c5c90-9418-42d4-821e-941b0e3658f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:34.927954 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.927883 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle" (OuterVolumeSpecName: "bundle") pod "d9e176e0-e101-48c5-bf58-849acc813fe0" (UID: "d9e176e0-e101-48c5-bf58-849acc813fe0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:34.929337 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.929312 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc" (OuterVolumeSpecName: "kube-api-access-6cfhc") pod "837c5c90-9418-42d4-821e-941b0e3658f9" (UID: "837c5c90-9418-42d4-821e-941b0e3658f9"). InnerVolumeSpecName "kube-api-access-6cfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:04:34.929421 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.929358 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5" (OuterVolumeSpecName: "kube-api-access-5dlz5") pod "d9e176e0-e101-48c5-bf58-849acc813fe0" (UID: "d9e176e0-e101-48c5-bf58-849acc813fe0"). InnerVolumeSpecName "kube-api-access-5dlz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:04:34.932961 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.932929 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util" (OuterVolumeSpecName: "util") pod "d9e176e0-e101-48c5-bf58-849acc813fe0" (UID: "d9e176e0-e101-48c5-bf58-849acc813fe0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:34.933128 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:34.933110 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util" (OuterVolumeSpecName: "util") pod "837c5c90-9418-42d4-821e-941b0e3658f9" (UID: "837c5c90-9418-42d4-821e-941b0e3658f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:04:35.027773 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027753 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.027773 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027773 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlz5\" (UniqueName: \"kubernetes.io/projected/d9e176e0-e101-48c5-bf58-849acc813fe0-kube-api-access-5dlz5\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.027902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027783 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.027902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027792 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837c5c90-9418-42d4-821e-941b0e3658f9-util\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.027902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027800 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfhc\" (UniqueName: \"kubernetes.io/projected/837c5c90-9418-42d4-821e-941b0e3658f9-kube-api-access-6cfhc\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.027902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.027807 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e176e0-e101-48c5-bf58-849acc813fe0-bundle\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:04:35.637157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.637132 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" Jul 31 23:04:35.637641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.637161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fvxfq" event={"ID":"d9e176e0-e101-48c5-bf58-849acc813fe0","Type":"ContainerDied","Data":"610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2"} Jul 31 23:04:35.637641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.637207 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610082f9ecde31e0860090fc98746218097361fb21147f96a02bc906eb0697e2" Jul 31 23:04:35.639043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.639015 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" event={"ID":"837c5c90-9418-42d4-821e-941b0e3658f9","Type":"ContainerDied","Data":"94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f"} Jul 31 23:04:35.639043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.639034 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9gt68" Jul 31 23:04:35.639236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.639059 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94b576132fcaea2172cfab2452c6cfda252253bfe0bea18818e6777062e4684f" Jul 31 23:04:35.640987 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.640956 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" event={"ID":"5306e963-6dfa-4d95-82e1-4895a08ac1ba","Type":"ContainerDied","Data":"4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14"} Jul 31 23:04:35.641100 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.641004 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca00a820eee98d8d18e1b8bcba12899cedccddf1c8fde401d95bcf448448e14" Jul 31 23:04:35.641100 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:35.640966 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cbchdg" Jul 31 23:04:57.186117 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186077 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl"] Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186374 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186389 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186409 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186418 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186428 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186437 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186450 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186458 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186479 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186488 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186497 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186503 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186510 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186515 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186521 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186526 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186535 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186544 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="pull" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186554 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eba0df4-71d2-4eee-a837-83157481d4ab" containerName="console" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186561 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eba0df4-71d2-4eee-a837-83157481d4ab" containerName="console" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186569 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186579 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186590 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186599 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186610 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186618 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="util" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186692 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eba0df4-71d2-4eee-a837-83157481d4ab" containerName="console" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186702 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e176e0-e101-48c5-bf58-849acc813fe0" containerName="extract" Jul 31 23:04:57.186690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186710 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5306e963-6dfa-4d95-82e1-4895a08ac1ba" containerName="extract" Jul 31 23:04:57.188200 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186720 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="837c5c90-9418-42d4-821e-941b0e3658f9" containerName="extract" Jul 31 23:04:57.188200 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.186731 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5bdf9a-ea47-445d-a67f-826681d9a70e" containerName="extract" Jul 31 23:04:57.189643 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.189622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.192514 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.192489 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 23:04:57.193904 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.193441 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 23:04:57.193904 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.193499 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9qxhs\"" Jul 31 23:04:57.193904 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.193643 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 31 23:04:57.193904 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.193776 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 31 23:04:57.200594 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.200570 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl"] Jul 31 23:04:57.282095 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.282064 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797zk\" (UniqueName: \"kubernetes.io/projected/fff8e9da-2fcf-4414-be73-6c42763e12cc-kube-api-access-797zk\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.282197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.282105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fff8e9da-2fcf-4414-be73-6c42763e12cc-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.282197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.282132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff8e9da-2fcf-4414-be73-6c42763e12cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.383191 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.383154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797zk\" (UniqueName: \"kubernetes.io/projected/fff8e9da-2fcf-4414-be73-6c42763e12cc-kube-api-access-797zk\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.383445 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.383226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fff8e9da-2fcf-4414-be73-6c42763e12cc-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.383445 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.383285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff8e9da-2fcf-4414-be73-6c42763e12cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.384051 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.384030 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fff8e9da-2fcf-4414-be73-6c42763e12cc-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.386027 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.386007 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fff8e9da-2fcf-4414-be73-6c42763e12cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.391571 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.391551 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-797zk\" (UniqueName: \"kubernetes.io/projected/fff8e9da-2fcf-4414-be73-6c42763e12cc-kube-api-access-797zk\") pod \"kuadrant-console-plugin-598f86468b-tl2nl\" (UID: \"fff8e9da-2fcf-4414-be73-6c42763e12cc\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.502436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.502404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" Jul 31 23:04:57.626100 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.626074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl"] Jul 31 23:04:57.627923 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:04:57.627882 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfff8e9da_2fcf_4414_be73_6c42763e12cc.slice/crio-15545cbae937e473cbb55fd1d3626c4aed3a9cbe8b77316832965a62031b5ef5 WatchSource:0}: Error finding container 15545cbae937e473cbb55fd1d3626c4aed3a9cbe8b77316832965a62031b5ef5: Status 404 returned error can't find the container with id 15545cbae937e473cbb55fd1d3626c4aed3a9cbe8b77316832965a62031b5ef5 Jul 31 23:04:57.740150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:04:57.740124 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" event={"ID":"fff8e9da-2fcf-4414-be73-6c42763e12cc","Type":"ContainerStarted","Data":"15545cbae937e473cbb55fd1d3626c4aed3a9cbe8b77316832965a62031b5ef5"} Jul 31 23:05:02.767792 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:02.767756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" event={"ID":"fff8e9da-2fcf-4414-be73-6c42763e12cc","Type":"ContainerStarted","Data":"d3b827402962a2bba21f8eb6df83e8c661c1a9914b8e0f36259a30cbfcaa3f2f"} Jul 31 23:05:02.783907 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:02.783813 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tl2nl" podStartSLOduration=1.212188348 podStartE2EDuration="5.783799061s" totalImagesPullingTime="4.571610713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:05:02.78325944 +0000 UTC m=+490.705527212" watchObservedRunningTime="2026-07-31 23:05:02.783799061 +0000 UTC m=+490.706066832" Jul 31 23:05:53.182073 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.181994 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:05:53.185249 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.185231 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.187692 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.187671 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 23:05:53.192913 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.192879 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:05:53.219472 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.219435 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:05:53.229215 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.229190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfhb\" (UniqueName: \"kubernetes.io/projected/230d3ca1-40fe-405e-ab1b-cb745647b7f1-kube-api-access-dgfhb\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.229304 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.229232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/230d3ca1-40fe-405e-ab1b-cb745647b7f1-config-file\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.329592 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.329557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfhb\" (UniqueName: \"kubernetes.io/projected/230d3ca1-40fe-405e-ab1b-cb745647b7f1-kube-api-access-dgfhb\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.329720 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.329609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/230d3ca1-40fe-405e-ab1b-cb745647b7f1-config-file\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.330242 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.330224 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/230d3ca1-40fe-405e-ab1b-cb745647b7f1-config-file\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.339115 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.339089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfhb\" (UniqueName: \"kubernetes.io/projected/230d3ca1-40fe-405e-ab1b-cb745647b7f1-kube-api-access-dgfhb\") pod \"limitador-limitador-85d9db4d5d-7rwtl\" (UID: \"230d3ca1-40fe-405e-ab1b-cb745647b7f1\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.497117 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.497081 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:53.628397 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.628346 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:05:53.630435 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:05:53.630401 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230d3ca1_40fe_405e_ab1b_cb745647b7f1.slice/crio-f65fd5149c3f12dc17c984faede9d311fb934f4b2acb63be49d09af489800d35 WatchSource:0}: Error finding container f65fd5149c3f12dc17c984faede9d311fb934f4b2acb63be49d09af489800d35: Status 404 returned error can't find the container with id f65fd5149c3f12dc17c984faede9d311fb934f4b2acb63be49d09af489800d35 Jul 31 23:05:53.682329 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.682295 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:05:53.686701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.686679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:53.689090 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.689071 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nfv87\"" Jul 31 23:05:53.694537 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.694517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:05:53.733024 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.732988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xnh\" (UniqueName: \"kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh\") pod \"authorino-b54df94b4-94t69\" (UID: \"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec\") " pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:53.834265 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.834159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xnh\" (UniqueName: \"kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh\") pod \"authorino-b54df94b4-94t69\" (UID: \"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec\") " pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:53.842467 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.842434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xnh\" (UniqueName: \"kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh\") pod \"authorino-b54df94b4-94t69\" (UID: \"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec\") " pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:53.996668 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:53.996633 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:54.023208 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:54.023175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" event={"ID":"230d3ca1-40fe-405e-ab1b-cb745647b7f1","Type":"ContainerStarted","Data":"f65fd5149c3f12dc17c984faede9d311fb934f4b2acb63be49d09af489800d35"} Jul 31 23:05:54.121999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:54.121973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:05:54.124265 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:05:54.124237 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c9d6cff_8db0_41bf_a0a1_f09293cb4cec.slice/crio-c8c6e006448aa6dfcaa2040bbc51d2353e6411aeb5e2684d013ba719b87b640d WatchSource:0}: Error finding container c8c6e006448aa6dfcaa2040bbc51d2353e6411aeb5e2684d013ba719b87b640d: Status 404 returned error can't find the container with id c8c6e006448aa6dfcaa2040bbc51d2353e6411aeb5e2684d013ba719b87b640d Jul 31 23:05:55.030923 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:55.030871 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-94t69" event={"ID":"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec","Type":"ContainerStarted","Data":"c8c6e006448aa6dfcaa2040bbc51d2353e6411aeb5e2684d013ba719b87b640d"} Jul 31 23:05:57.044696 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:57.044660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" event={"ID":"230d3ca1-40fe-405e-ab1b-cb745647b7f1","Type":"ContainerStarted","Data":"365c4e016dc0b6c41cc62f76aeb291f8aa0aa4fa3ab8f59134470281e07af0cd"} Jul 31 23:05:57.045125 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:57.044741 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:05:57.062074 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:57.062028 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" podStartSLOduration=1.580321578 podStartE2EDuration="4.062014735s" totalImagesPullingTime="2.481693157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:05:57.061048387 +0000 UTC m=+544.983316159" watchObservedRunningTime="2026-07-31 23:05:57.062014735 +0000 UTC m=+544.984282530" Jul 31 23:05:57.651267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:57.651184 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:05:58.051444 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:58.051405 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-94t69" event={"ID":"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec","Type":"ContainerStarted","Data":"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7"} Jul 31 23:05:58.067178 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:58.067131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-94t69" podStartSLOduration=1.9237473280000001 podStartE2EDuration="5.067114397s" totalImagesPullingTime="3.143367069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:05:58.065736026 +0000 UTC m=+545.988003810" watchObservedRunningTime="2026-07-31 23:05:58.067114397 +0000 UTC m=+545.989382167" Jul 31 23:05:59.055970 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:59.055925 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-94t69" podUID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" containerName="authorino" containerID="cri-o://6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7" gracePeriod=30 Jul 31 23:05:59.302979 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:59.302949 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:05:59.385278 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:59.385211 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8xnh\" (UniqueName: \"kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh\") pod \"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec\" (UID: \"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec\") " Jul 31 23:05:59.387269 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:59.387243 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh" (OuterVolumeSpecName: "kube-api-access-k8xnh") pod "2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" (UID: "2c9d6cff-8db0-41bf-a0a1-f09293cb4cec"). InnerVolumeSpecName "kube-api-access-k8xnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:05:59.486021 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:05:59.485986 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xnh\" (UniqueName: \"kubernetes.io/projected/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec-kube-api-access-k8xnh\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:06:00.063961 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.063919 2570 generic.go:356] "Generic (PLEG): container finished" podID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" containerID="6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7" exitCode=0 Jul 31 23:06:00.064426 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.063963 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-94t69" Jul 31 23:06:00.064426 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.063996 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-94t69" event={"ID":"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec","Type":"ContainerDied","Data":"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7"} Jul 31 23:06:00.064426 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.064037 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-94t69" event={"ID":"2c9d6cff-8db0-41bf-a0a1-f09293cb4cec","Type":"ContainerDied","Data":"c8c6e006448aa6dfcaa2040bbc51d2353e6411aeb5e2684d013ba719b87b640d"} Jul 31 23:06:00.064426 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.064053 2570 scope.go:117] "RemoveContainer" containerID="6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7" Jul 31 23:06:00.072775 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.072757 2570 scope.go:117] "RemoveContainer" containerID="6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7" Jul 31 23:06:00.073053 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:06:00.073035 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7\": container with ID starting with 6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7 not found: ID does not exist" containerID="6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7" Jul 31 23:06:00.073113 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.073062 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7"} err="failed to get container status \"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7\": rpc error: code = NotFound desc = could not find container \"6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7\": container with ID starting with 6bc625847e088e9f22536c88d7e3046322514dc5f76c725675d546768b5a10b7 not found: ID does not exist" Jul 31 23:06:00.088947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.085081 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:06:00.090693 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.090345 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-94t69"] Jul 31 23:06:00.537987 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:00.537951 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" path="/var/lib/kubelet/pods/2c9d6cff-8db0-41bf-a0a1-f09293cb4cec/volumes" Jul 31 23:06:08.052942 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:08.052906 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl" Jul 31 23:06:52.481850 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:52.481806 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:06:52.484305 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:52.484284 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:06:52.486137 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:52.486118 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:06:52.488764 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:06:52.488745 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:07:06.011363 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:06.011330 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:07:19.486103 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:19.486025 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:07:22.186564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:22.186519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:07:29.504763 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:29.504727 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:07:36.389777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:36.389730 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:07:59.889299 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:07:59.889257 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:08:16.892449 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:08:16.892414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:09:07.227129 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.227041 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dd6b4dfd5-sc5fm"] Jul 31 23:09:07.227575 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.227380 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" containerName="authorino" Jul 31 23:09:07.227575 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.227393 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" containerName="authorino" Jul 31 23:09:07.227575 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.227446 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9d6cff-8db0-41bf-a0a1-f09293cb4cec" containerName="authorino" Jul 31 23:09:07.230176 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.230160 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.233797 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.233770 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 31 23:09:07.233797 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.233795 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nfv87\"" Jul 31 23:09:07.237686 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.237661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dd6b4dfd5-sc5fm"] Jul 31 23:09:07.248955 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.248931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3fd9344b-7bb6-4f0a-8a21-94048704e45d-tls-cert\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.249080 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.249016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsrk\" (UniqueName: \"kubernetes.io/projected/3fd9344b-7bb6-4f0a-8a21-94048704e45d-kube-api-access-rpsrk\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.350286 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.350250 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3fd9344b-7bb6-4f0a-8a21-94048704e45d-tls-cert\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.350447 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.350331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsrk\" (UniqueName: \"kubernetes.io/projected/3fd9344b-7bb6-4f0a-8a21-94048704e45d-kube-api-access-rpsrk\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.352687 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.352666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3fd9344b-7bb6-4f0a-8a21-94048704e45d-tls-cert\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.358169 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.358144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsrk\" (UniqueName: \"kubernetes.io/projected/3fd9344b-7bb6-4f0a-8a21-94048704e45d-kube-api-access-rpsrk\") pod \"authorino-5dd6b4dfd5-sc5fm\" (UID: \"3fd9344b-7bb6-4f0a-8a21-94048704e45d\") " pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.539791 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.539712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" Jul 31 23:09:07.671361 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.671324 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dd6b4dfd5-sc5fm"] Jul 31 23:09:07.674030 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:09:07.674003 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fd9344b_7bb6_4f0a_8a21_94048704e45d.slice/crio-c6eeb835e87f4dba1940dca1915436de50785b91cfd43ede84378ab7aee58632 WatchSource:0}: Error finding container c6eeb835e87f4dba1940dca1915436de50785b91cfd43ede84378ab7aee58632: Status 404 returned error can't find the container with id c6eeb835e87f4dba1940dca1915436de50785b91cfd43ede84378ab7aee58632 Jul 31 23:09:07.675614 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.675591 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:09:07.948423 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:07.948383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" event={"ID":"3fd9344b-7bb6-4f0a-8a21-94048704e45d","Type":"ContainerStarted","Data":"c6eeb835e87f4dba1940dca1915436de50785b91cfd43ede84378ab7aee58632"} Jul 31 23:09:08.955580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:08.955531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" event={"ID":"3fd9344b-7bb6-4f0a-8a21-94048704e45d","Type":"ContainerStarted","Data":"b6a84c0455ca072a770d1cd278a50715c4596384eae61cdbb1940824ca6b90ba"} Jul 31 23:09:08.976107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:08.976050 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dd6b4dfd5-sc5fm" podStartSLOduration=1.3930365550000001 podStartE2EDuration="1.976035719s" totalImagesPullingTime="582.999164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:09:08.974686888 +0000 UTC m=+736.896954659" watchObservedRunningTime="2026-07-31 23:09:08.976035719 +0000 UTC m=+736.898303489" Jul 31 23:09:38.799952 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:38.799896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:09:48.405058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:09:48.404990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:10:01.135213 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:10:01.135174 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:11:39.610050 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:39.609978 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:11:48.897798 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:48.897684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:11:52.517806 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:52.517766 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:11:52.521027 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:52.520995 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:11:52.522676 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:52.522648 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:11:52.525572 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:11:52.525547 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:12:48.603992 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:12:48.603957 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:12:55.407769 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:12:55.407732 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:10.791159 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:10.791120 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:16.607852 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:16.607759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:26.202040 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:26.202001 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:34.801506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:34.801471 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:46.605413 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:46.605377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:13:51.705814 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:13:51.705779 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:14:00.698865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:14:00.698803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:14:08.998169 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:14:08.998106 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:14:21.508516 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:14:21.508456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:14:32.201881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:14:32.201799 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:14:54.905755 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:14:54.905614 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:05.097625 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:05.097585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:10.997425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:10.997389 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:18.698302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:18.698266 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:26.231682 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:26.231646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:31.296211 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:31.296177 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:37.296542 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:37.296507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:15:43.994935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:15:43.994895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:00.098360 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:00.098327 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:05.493527 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:05.493489 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:15.504904 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:15.504807 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:20.799495 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:20.799461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:27.100217 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:27.100146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:35.700838 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:35.700772 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:40.693464 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:40.693397 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:51.607931 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:51.607873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:16:52.565245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:52.565182 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:16:52.568701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:52.568662 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:16:52.571003 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:52.570975 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:16:52.574257 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:16:52.574226 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:19:07.290210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:19:07.290174 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:19:14.290341 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:19:14.290301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:19:36.390255 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:19:36.390216 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:00.692663 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:00.692621 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:07.289977 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:07.289929 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:16.489137 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:16.489105 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:22.289407 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:22.289371 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:32.996098 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:32.996059 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:42.689895 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:42.689861 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:47.684087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:47.684006 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:20:52.593032 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:20:52.592976 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:03.196756 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:03.196707 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:09.494002 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:09.493960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:19.592009 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:19.591970 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:42.191205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:42.191164 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:49.191201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:49.191162 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:21:52.600612 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:52.600580 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:21:52.604769 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:52.604748 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:21:52.604914 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:52.604807 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:21:52.609274 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:21:52.609253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:22:01.298128 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:01.298069 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:22:18.098908 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:18.098803 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:22:37.489626 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:37.489588 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:22:47.788643 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:47.788607 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:22:53.101235 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:53.101203 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:22:59.291008 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:22:59.290969 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:09.093058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:09.093025 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:15.191363 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:15.191325 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:21.900327 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:21.900287 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:28.485519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:28.485478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:34.089062 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:34.089029 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:43.591807 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:43.591769 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:23:49.785100 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:23:49.785017 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:05.892441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:05.892403 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:12.693116 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:12.693077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:24.391509 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:24.391472 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:31.284966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:31.284930 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:37.488542 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:37.488497 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:43.286228 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:43.286192 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:49.589118 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:49.589081 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:24:58.991223 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:24:58.991187 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:06.195515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:06.195474 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:15.989960 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:15.989883 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:22.093860 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:22.093814 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:32.885086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:32.885050 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:47.898630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:47.898592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:25:53.989248 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:25:53.989211 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:00.288029 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:00.287994 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:11.388971 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:11.388930 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:19.693342 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:19.693312 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:25.193917 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:25.193884 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:38.592594 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:38.592558 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:26:52.630948 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:52.630917 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:26:52.635179 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:52.635151 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:26:52.637713 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:52.637693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:26:52.641760 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:52.641744 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:26:59.963094 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:59.963054 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc"] Jul 31 23:26:59.966369 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:59.966349 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:26:59.968992 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:59.968970 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:26:59.969112 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:59.969045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:26:59.972772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:26:59.972751 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc"] Jul 31 23:27:00.058610 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.058583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc\" (UID: \"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:00.159639 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.159615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc\" (UID: \"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:00.160217 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.160193 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc\" (UID: \"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:00.278919 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.278852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:00.405715 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.405688 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc"] Jul 31 23:27:00.407195 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:27:00.407168 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ccfd16_bb4d_4b04_b3f6_3d4438c9f534.slice/crio-20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89 WatchSource:0}: Error finding container 20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89: Status 404 returned error can't find the container with id 20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89 Jul 31 23:27:00.408888 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:00.408869 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:27:01.408106 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:01.408016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" event={"ID":"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534","Type":"ContainerStarted","Data":"20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89"} Jul 31 23:27:03.419808 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:03.419774 2570 generic.go:356] "Generic (PLEG): container finished" podID="56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" containerID="cc25f8a238b94586d0306a4107a13a94fd52a536a240f782f237f9b2fa1a1554" exitCode=0 Jul 31 23:27:03.420128 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:03.419831 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" event={"ID":"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534","Type":"ContainerDied","Data":"cc25f8a238b94586d0306a4107a13a94fd52a536a240f782f237f9b2fa1a1554"} Jul 31 23:27:04.555003 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:04.554980 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:04.695718 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:04.695694 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca\") pod \"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534\" (UID: \"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534\") " Jul 31 23:27:04.696030 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:04.696009 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" (UID: "56ccfd16-bb4d-4b04-b3f6-3d4438c9f534"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:27:04.796487 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:04.796462 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:27:05.430659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:05.430635 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" Jul 31 23:27:05.430877 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:05.430627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc" event={"ID":"56ccfd16-bb4d-4b04-b3f6-3d4438c9f534","Type":"ContainerDied","Data":"20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89"} Jul 31 23:27:05.430877 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:05.430757 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b0469cd30a950aec6350e2e6ad68017ae7cf361c5516cd113c134fdd9f3f89" Jul 31 23:27:16.330985 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:16.330948 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc"] Jul 31 23:27:16.333195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:16.333172 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fec77b77-p5gkc"] Jul 31 23:27:16.537913 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:16.537878 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" path="/var/lib/kubelet/pods/56ccfd16-bb4d-4b04-b3f6-3d4438c9f534/volumes" Jul 31 23:27:19.286738 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:19.286706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:27:26.598630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:26.598594 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:27:44.796218 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.796182 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d"] Jul 31 23:27:44.796611 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.796483 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" containerName="revoke-keys" Jul 31 23:27:44.796611 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.796494 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" containerName="revoke-keys" Jul 31 23:27:44.796611 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.796573 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ccfd16-bb4d-4b04-b3f6-3d4438c9f534" containerName="revoke-keys" Jul 31 23:27:44.803874 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.803847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:44.806701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.806339 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d"] Jul 31 23:27:44.806701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.806488 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:27:44.807056 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.806796 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:27:44.878709 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.878683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d\" (UID: \"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:44.979077 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.979049 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d\" (UID: \"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:44.979634 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:44.979614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d\" (UID: \"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:45.116564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:45.116512 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:45.239634 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:45.239607 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d"] Jul 31 23:27:45.241503 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:27:45.241474 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae2462d_c1a0_4baa_bbc0_c96a41690bd2.slice/crio-cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa WatchSource:0}: Error finding container cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa: Status 404 returned error can't find the container with id cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa Jul 31 23:27:45.642518 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:45.642490 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" event={"ID":"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2","Type":"ContainerStarted","Data":"efa2943711f1fa9e025fbb7537e6e8c6708961f405ea8dc4e40517b8cd48d5bb"} Jul 31 23:27:45.642634 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:45.642541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" event={"ID":"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2","Type":"ContainerStarted","Data":"cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa"} Jul 31 23:27:45.657685 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:45.657639 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" podStartSLOduration=1.657624386 podStartE2EDuration="1.657624386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:27:45.655504052 +0000 UTC m=+1853.577771823" watchObservedRunningTime="2026-07-31 23:27:45.657624386 +0000 UTC m=+1853.579892157" Jul 31 23:27:46.648343 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:46.648302 2570 generic.go:356] "Generic (PLEG): container finished" podID="2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" containerID="efa2943711f1fa9e025fbb7537e6e8c6708961f405ea8dc4e40517b8cd48d5bb" exitCode=0 Jul 31 23:27:46.648711 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:46.648391 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" event={"ID":"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2","Type":"ContainerDied","Data":"efa2943711f1fa9e025fbb7537e6e8c6708961f405ea8dc4e40517b8cd48d5bb"} Jul 31 23:27:47.780616 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:47.780590 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:47.902736 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:47.902704 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca\") pod \"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2\" (UID: \"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2\") " Jul 31 23:27:47.903075 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:47.903052 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" (UID: "2ae2462d-c1a0-4baa-bbc0-c96a41690bd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:27:48.003701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:48.003677 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:27:48.659243 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:48.659217 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" Jul 31 23:27:48.659380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:48.659213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d" event={"ID":"2ae2462d-c1a0-4baa-bbc0-c96a41690bd2","Type":"ContainerDied","Data":"cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa"} Jul 31 23:27:48.659380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:48.659341 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb15eff4ae36fddcdec3dfce23b97bee2e25921c58fcb813be162386bebe43aa" Jul 31 23:27:50.576590 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:50.576557 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d"] Jul 31 23:27:50.579898 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:50.579871 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13204f82-mgv5d"] Jul 31 23:27:52.538906 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:52.538871 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" path="/var/lib/kubelet/pods/2ae2462d-c1a0-4baa-bbc0-c96a41690bd2/volumes" Jul 31 23:27:53.286360 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:27:53.286326 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:28:02.997618 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:02.997585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:28:19.608499 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.608429 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d"] Jul 31 23:28:19.608862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.608733 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" containerName="revoke-keys" Jul 31 23:28:19.608862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.608743 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" containerName="revoke-keys" Jul 31 23:28:19.608862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.608791 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae2462d-c1a0-4baa-bbc0-c96a41690bd2" containerName="revoke-keys" Jul 31 23:28:19.611501 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.611482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:19.613887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.613865 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:28:19.614007 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.613894 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:28:19.621001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.620977 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d"] Jul 31 23:28:19.732550 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.732517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-86538af5-gbl7d\" (UID: \"69988154-2dc0-4581-81f3-63d1855d0ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:19.833345 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.833308 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-86538af5-gbl7d\" (UID: \"69988154-2dc0-4581-81f3-63d1855d0ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:19.833928 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.833908 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-86538af5-gbl7d\" (UID: \"69988154-2dc0-4581-81f3-63d1855d0ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:19.922376 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:19.922319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:20.050584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:20.050557 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d"] Jul 31 23:28:20.052259 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:28:20.052237 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69988154_2dc0_4581_81f3_63d1855d0ac1.slice/crio-43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d WatchSource:0}: Error finding container 43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d: Status 404 returned error can't find the container with id 43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d Jul 31 23:28:20.820918 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:20.820885 2570 generic.go:356] "Generic (PLEG): container finished" podID="69988154-2dc0-4581-81f3-63d1855d0ac1" containerID="70e4e733e48568174a1b044ca7df61229cf509c5211ccaee15a05cfac103e28f" exitCode=0 Jul 31 23:28:20.821245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:20.820952 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" event={"ID":"69988154-2dc0-4581-81f3-63d1855d0ac1","Type":"ContainerDied","Data":"70e4e733e48568174a1b044ca7df61229cf509c5211ccaee15a05cfac103e28f"} Jul 31 23:28:20.821245 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:20.820997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" event={"ID":"69988154-2dc0-4581-81f3-63d1855d0ac1","Type":"ContainerStarted","Data":"43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d"} Jul 31 23:28:21.951087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:21.951065 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:22.051232 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.051207 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca\") pod \"69988154-2dc0-4581-81f3-63d1855d0ac1\" (UID: \"69988154-2dc0-4581-81f3-63d1855d0ac1\") " Jul 31 23:28:22.051547 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.051526 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69988154-2dc0-4581-81f3-63d1855d0ac1" (UID: "69988154-2dc0-4581-81f3-63d1855d0ac1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:28:22.152628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.152580 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69988154-2dc0-4581-81f3-63d1855d0ac1-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:28:22.834326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.834250 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" Jul 31 23:28:22.834326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.834266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d" event={"ID":"69988154-2dc0-4581-81f3-63d1855d0ac1","Type":"ContainerDied","Data":"43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d"} Jul 31 23:28:22.834326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:22.834311 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43dee842863f823ad9d27b5738aace60eaac59da5934edcec390d7e21ecb421d" Jul 31 23:28:25.636858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:25.636812 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d"] Jul 31 23:28:25.639787 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:25.639762 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-86538af5-gbl7d"] Jul 31 23:28:26.538274 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:26.538235 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69988154-2dc0-4581-81f3-63d1855d0ac1" path="/var/lib/kubelet/pods/69988154-2dc0-4581-81f3-63d1855d0ac1/volumes" Jul 31 23:28:28.598802 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:28.598773 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:28:38.697743 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:38.697705 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:28:54.968225 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.968188 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8"] Jul 31 23:28:54.968650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.968529 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69988154-2dc0-4581-81f3-63d1855d0ac1" containerName="revoke-keys" Jul 31 23:28:54.968650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.968545 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69988154-2dc0-4581-81f3-63d1855d0ac1" containerName="revoke-keys" Jul 31 23:28:54.968650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.968604 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="69988154-2dc0-4581-81f3-63d1855d0ac1" containerName="revoke-keys" Jul 31 23:28:54.972640 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.972620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:54.976314 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.976287 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:28:54.976488 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.976291 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:28:54.978298 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:54.978277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8"] Jul 31 23:28:55.088169 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.088137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7056468c-rhgp8\" (UID: \"32d4946b-19a6-4586-8c1f-840e5f141c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:55.188401 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.188376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7056468c-rhgp8\" (UID: \"32d4946b-19a6-4586-8c1f-840e5f141c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:55.188985 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.188965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7056468c-rhgp8\" (UID: \"32d4946b-19a6-4586-8c1f-840e5f141c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:55.285688 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.285625 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:55.415103 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.412365 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8"] Jul 31 23:28:55.993871 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.993835 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" event={"ID":"32d4946b-19a6-4586-8c1f-840e5f141c1d","Type":"ContainerStarted","Data":"519b3c2309cbdd61f71f4ab6a62fb1368aa85b40d4060b6ac38d245a0340cf26"} Jul 31 23:28:55.994288 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:55.993892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" event={"ID":"32d4946b-19a6-4586-8c1f-840e5f141c1d","Type":"ContainerStarted","Data":"ef115e5f45e981500b3037f26fedf3cc7a54b8bea23fadb84b0b416840cbf652"} Jul 31 23:28:56.008021 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:56.007979 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" podStartSLOduration=2.007965591 podStartE2EDuration="2.007965591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:28:56.007021849 +0000 UTC m=+1923.929289619" watchObservedRunningTime="2026-07-31 23:28:56.007965591 +0000 UTC m=+1923.930233340" Jul 31 23:28:56.999805 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:56.999770 2570 generic.go:356] "Generic (PLEG): container finished" podID="32d4946b-19a6-4586-8c1f-840e5f141c1d" containerID="519b3c2309cbdd61f71f4ab6a62fb1368aa85b40d4060b6ac38d245a0340cf26" exitCode=0 Jul 31 23:28:57.000232 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:56.999865 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" event={"ID":"32d4946b-19a6-4586-8c1f-840e5f141c1d","Type":"ContainerDied","Data":"519b3c2309cbdd61f71f4ab6a62fb1368aa85b40d4060b6ac38d245a0340cf26"} Jul 31 23:28:58.135981 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:58.135957 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:58.311562 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:58.311488 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca\") pod \"32d4946b-19a6-4586-8c1f-840e5f141c1d\" (UID: \"32d4946b-19a6-4586-8c1f-840e5f141c1d\") " Jul 31 23:28:58.311841 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:58.311799 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32d4946b-19a6-4586-8c1f-840e5f141c1d" (UID: "32d4946b-19a6-4586-8c1f-840e5f141c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:28:58.412448 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:58.412418 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d4946b-19a6-4586-8c1f-840e5f141c1d-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:28:59.010666 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:59.010638 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" Jul 31 23:28:59.010850 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:59.010675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8" event={"ID":"32d4946b-19a6-4586-8c1f-840e5f141c1d","Type":"ContainerDied","Data":"ef115e5f45e981500b3037f26fedf3cc7a54b8bea23fadb84b0b416840cbf652"} Jul 31 23:28:59.010850 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:28:59.010720 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef115e5f45e981500b3037f26fedf3cc7a54b8bea23fadb84b0b416840cbf652" Jul 31 23:29:01.863337 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:29:01.863303 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8"] Jul 31 23:29:01.866733 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:29:01.866706 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7056468c-rhgp8"] Jul 31 23:29:02.544425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:29:02.544393 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d4946b-19a6-4586-8c1f-840e5f141c1d" path="/var/lib/kubelet/pods/32d4946b-19a6-4586-8c1f-840e5f141c1d/volumes" Jul 31 23:29:04.490260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:29:04.490225 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:29:15.986572 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:29:15.986534 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:30:00.130501 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.130424 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5"] Jul 31 23:30:00.130973 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.130728 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d4946b-19a6-4586-8c1f-840e5f141c1d" containerName="revoke-keys" Jul 31 23:30:00.130973 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.130739 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d4946b-19a6-4586-8c1f-840e5f141c1d" containerName="revoke-keys" Jul 31 23:30:00.130973 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.130801 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d4946b-19a6-4586-8c1f-840e5f141c1d" containerName="revoke-keys" Jul 31 23:30:00.133720 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.133701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" Jul 31 23:30:00.136999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.136975 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:30:00.140065 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.140042 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5"] Jul 31 23:30:00.145554 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.145535 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" Jul 31 23:30:00.270688 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.270664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5"] Jul 31 23:30:00.273260 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:30:00.273235 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900dadec_966d_499b_b786_e7ddc9e449bb.slice/crio-9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a WatchSource:0}: Error finding container 9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a: Status 404 returned error can't find the container with id 9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a Jul 31 23:30:00.331575 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:00.331543 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" event={"ID":"900dadec-966d-499b-b786-e7ddc9e449bb","Type":"ContainerStarted","Data":"9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a"} Jul 31 23:30:01.337887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:01.337856 2570 generic.go:356] "Generic (PLEG): container finished" podID="900dadec-966d-499b-b786-e7ddc9e449bb" containerID="74131bd9b27e30b8729be74e95de45ac2ff7f5eb0618e9694e77a3b419513c3a" exitCode=0 Jul 31 23:30:01.338241 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:01.337947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" event={"ID":"900dadec-966d-499b-b786-e7ddc9e449bb","Type":"ContainerDied","Data":"74131bd9b27e30b8729be74e95de45ac2ff7f5eb0618e9694e77a3b419513c3a"} Jul 31 23:30:02.468291 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:02.468269 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" Jul 31 23:30:03.349732 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:03.349697 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" Jul 31 23:30:03.349952 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:03.349749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759010-j4xw5" event={"ID":"900dadec-966d-499b-b786-e7ddc9e449bb","Type":"ContainerDied","Data":"9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a"} Jul 31 23:30:03.349952 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:03.349807 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e61066fa8411b2477a7b8f50eab45edb2230064b0053d1157e6629afc15b61a" Jul 31 23:30:43.794370 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:43.794329 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:30:53.790783 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:30:53.790741 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:31:04.297777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:04.297740 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:31:32.541347 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.541300 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg"] Jul 31 23:31:32.541847 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.541606 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900dadec-966d-499b-b786-e7ddc9e449bb" containerName="cleanup" Jul 31 23:31:32.541847 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.541617 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="900dadec-966d-499b-b786-e7ddc9e449bb" containerName="cleanup" Jul 31 23:31:32.541847 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.541678 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="900dadec-966d-499b-b786-e7ddc9e449bb" containerName="cleanup" Jul 31 23:31:32.544535 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.544518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:32.547081 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.547056 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:31:32.547197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.547163 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:31:32.551161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.551136 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg"] Jul 31 23:31:32.653314 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.653272 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg\" (UID: \"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:32.754142 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.754103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg\" (UID: \"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:32.754650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.754631 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg\" (UID: \"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:32.855723 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.855667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:32.983803 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:32.983777 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg"] Jul 31 23:31:32.985728 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:31:32.985701 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d5a1c9_a83b_4d84_9ba2_6a7c7df7a559.slice/crio-95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1 WatchSource:0}: Error finding container 95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1: Status 404 returned error can't find the container with id 95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1 Jul 31 23:31:33.817759 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:33.817683 2570 generic.go:356] "Generic (PLEG): container finished" podID="03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" containerID="57165e593276f3a2abf45b08dc7c636e3a394ce2fa398e3b674ec748ad90e07f" exitCode=0 Jul 31 23:31:33.818103 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:33.817769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" event={"ID":"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559","Type":"ContainerDied","Data":"57165e593276f3a2abf45b08dc7c636e3a394ce2fa398e3b674ec748ad90e07f"} Jul 31 23:31:33.818103 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:33.817849 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" event={"ID":"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559","Type":"ContainerStarted","Data":"95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1"} Jul 31 23:31:34.946984 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:34.946963 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:35.072163 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.072140 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca\") pod \"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559\" (UID: \"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559\") " Jul 31 23:31:35.072485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.072461 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" (UID: "03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:31:35.173514 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.173463 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:31:35.829076 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.829048 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" Jul 31 23:31:35.829236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.829101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg" event={"ID":"03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559","Type":"ContainerDied","Data":"95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1"} Jul 31 23:31:35.829236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:35.829153 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95586042e78d96ab9642632377e60966a8e0a06b42e0607a1a760106f290d3f1" Jul 31 23:31:43.574227 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:43.574190 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg"] Jul 31 23:31:43.575967 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:43.575944 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-124aaef2-b4zpg"] Jul 31 23:31:44.543125 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:44.543091 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" path="/var/lib/kubelet/pods/03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559/volumes" Jul 31 23:31:45.684496 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:45.684462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:31:52.661542 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:52.661508 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:31:52.665690 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:52.665665 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:31:52.670121 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:52.670102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:31:52.674077 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:31:52.674060 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:32:21.194394 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:32:21.194359 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:33:52.708200 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:33:52.708171 2570 scope.go:117] "RemoveContainer" containerID="cc25f8a238b94586d0306a4107a13a94fd52a536a240f782f237f9b2fa1a1554" Jul 31 23:33:52.716340 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:33:52.716321 2570 scope.go:117] "RemoveContainer" containerID="efa2943711f1fa9e025fbb7537e6e8c6708961f405ea8dc4e40517b8cd48d5bb" Jul 31 23:34:07.595701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:34:07.595664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:34:16.003272 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:34:16.003192 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:34:52.737653 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:34:52.737621 2570 scope.go:117] "RemoveContainer" containerID="70e4e733e48568174a1b044ca7df61229cf509c5211ccaee15a05cfac103e28f" Jul 31 23:34:52.995635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:34:52.995561 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:35:06.601118 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:06.601073 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:35:16.880999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.880955 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5"] Jul 31 23:35:16.881375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.881288 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" containerName="revoke-keys" Jul 31 23:35:16.881375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.881299 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" containerName="revoke-keys" Jul 31 23:35:16.881375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.881356 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d5a1c9-a83b-4d84-9ba2-6a7c7df7a559" containerName="revoke-keys" Jul 31 23:35:16.884275 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.884258 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:16.886829 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.886796 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:35:16.886916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.886808 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:35:16.892217 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.892182 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5"] Jul 31 23:35:16.936584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:16.936554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5\" (UID: \"3979ea2e-f969-416c-9f8f-3a21995fd9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:17.037322 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.037285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5\" (UID: \"3979ea2e-f969-416c-9f8f-3a21995fd9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:17.037891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.037874 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5\" (UID: \"3979ea2e-f969-416c-9f8f-3a21995fd9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:17.202506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.202482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:17.331474 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.331378 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5"] Jul 31 23:35:17.333721 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:35:17.333686 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3979ea2e_f969_416c_9f8f_3a21995fd9d1.slice/crio-fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042 WatchSource:0}: Error finding container fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042: Status 404 returned error can't find the container with id fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042 Jul 31 23:35:17.939162 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.939127 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" event={"ID":"3979ea2e-f969-416c-9f8f-3a21995fd9d1","Type":"ContainerStarted","Data":"55609de1113d27be456c9912894e7223e3499c4a0877126b0f26f6fc9098f369"} Jul 31 23:35:17.939543 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.939181 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" event={"ID":"3979ea2e-f969-416c-9f8f-3a21995fd9d1","Type":"ContainerStarted","Data":"fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042"} Jul 31 23:35:17.953788 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:17.953741 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" podStartSLOduration=1.953726317 podStartE2EDuration="1.953726317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:35:17.952648644 +0000 UTC m=+2305.874916406" watchObservedRunningTime="2026-07-31 23:35:17.953726317 +0000 UTC m=+2305.875994088" Jul 31 23:35:18.945162 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:18.945126 2570 generic.go:356] "Generic (PLEG): container finished" podID="3979ea2e-f969-416c-9f8f-3a21995fd9d1" containerID="55609de1113d27be456c9912894e7223e3499c4a0877126b0f26f6fc9098f369" exitCode=0 Jul 31 23:35:18.945540 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:18.945188 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" event={"ID":"3979ea2e-f969-416c-9f8f-3a21995fd9d1","Type":"ContainerDied","Data":"55609de1113d27be456c9912894e7223e3499c4a0877126b0f26f6fc9098f369"} Jul 31 23:35:20.074581 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.074554 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:20.161260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.161230 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca\") pod \"3979ea2e-f969-416c-9f8f-3a21995fd9d1\" (UID: \"3979ea2e-f969-416c-9f8f-3a21995fd9d1\") " Jul 31 23:35:20.161540 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.161519 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3979ea2e-f969-416c-9f8f-3a21995fd9d1" (UID: "3979ea2e-f969-416c-9f8f-3a21995fd9d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:35:20.262688 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.262630 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3979ea2e-f969-416c-9f8f-3a21995fd9d1-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:35:20.959035 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.959003 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" event={"ID":"3979ea2e-f969-416c-9f8f-3a21995fd9d1","Type":"ContainerDied","Data":"fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042"} Jul 31 23:35:20.959197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.959059 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcac45ad3f742a72d8dc5283fc8ad55fa05bbb325c8340ece8e4ca3584189042" Jul 31 23:35:20.959197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:20.959018 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5" Jul 31 23:35:29.985943 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:29.985902 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:35:32.542628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:32.542597 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5"] Jul 31 23:35:32.544698 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:32.544677 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0a726cba-dv8g5"] Jul 31 23:35:34.538004 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:34.537965 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3979ea2e-f969-416c-9f8f-3a21995fd9d1" path="/var/lib/kubelet/pods/3979ea2e-f969-416c-9f8f-3a21995fd9d1/volumes" Jul 31 23:35:35.789499 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:35.789464 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:35:52.759638 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:35:52.759555 2570 scope.go:117] "RemoveContainer" containerID="519b3c2309cbdd61f71f4ab6a62fb1368aa85b40d4060b6ac38d245a0340cf26" Jul 31 23:36:25.395790 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:25.395751 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:36:37.599628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:37.599592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:36:45.291510 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:45.291475 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:36:52.692563 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:52.692534 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:36:52.696886 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:52.696864 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:36:52.703003 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:52.702986 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:36:52.706791 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:36:52.706774 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:37:36.393753 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:37:36.393721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:37:52.788420 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:37:52.788386 2570 scope.go:117] "RemoveContainer" containerID="57165e593276f3a2abf45b08dc7c636e3a394ce2fa398e3b674ec748ad90e07f" Jul 31 23:38:03.891341 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:03.891305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:38:33.382661 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.382630 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:38:33.383153 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.382938 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3979ea2e-f969-416c-9f8f-3a21995fd9d1" containerName="revoke-keys" Jul 31 23:38:33.383153 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.382949 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3979ea2e-f969-416c-9f8f-3a21995fd9d1" containerName="revoke-keys" Jul 31 23:38:33.383153 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.383012 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3979ea2e-f969-416c-9f8f-3a21995fd9d1" containerName="revoke-keys" Jul 31 23:38:33.385742 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.385721 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.389452 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.389421 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-02002b\"/\"kube-root-ca.crt\"" Jul 31 23:38:33.389590 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.389494 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-02002b\"/\"default-dockercfg-x5w55\"" Jul 31 23:38:33.389590 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.389424 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-02002b\"/\"test-model-02002b-kserve-self-signed-certs\"" Jul 31 23:38:33.389712 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.389615 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-02002b\"/\"openshift-service-ca.crt\"" Jul 31 23:38:33.398235 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.398211 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:38:33.516425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.516589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.516589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.516589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.516589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.516589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.516544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7wq\" (UniqueName: \"kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617755 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.617891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.617784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7wq\" (UniqueName: \"kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.618066 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.618044 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.618187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.618165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.618254 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.618225 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.619983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.619959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.620254 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.620234 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.626045 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.626023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7wq\" (UniqueName: \"kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq\") pod \"test-model-02002b-kserve-86f8df6745-zwjht\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.694919 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.694898 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:33.823374 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.823344 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:38:33.825067 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:38:33.825035 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11037542_cc8e_4b76_9d41_90efd4829a01.slice/crio-f3b96afd88d7cbe51764f4fcc3e8ee8b1e0a67e4bd9efbc35649e9722ec0c862 WatchSource:0}: Error finding container f3b96afd88d7cbe51764f4fcc3e8ee8b1e0a67e4bd9efbc35649e9722ec0c862: Status 404 returned error can't find the container with id f3b96afd88d7cbe51764f4fcc3e8ee8b1e0a67e4bd9efbc35649e9722ec0c862 Jul 31 23:38:33.827036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.827019 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:38:33.917685 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:33.917653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" event={"ID":"11037542-cc8e-4b76-9d41-90efd4829a01","Type":"ContainerStarted","Data":"f3b96afd88d7cbe51764f4fcc3e8ee8b1e0a67e4bd9efbc35649e9722ec0c862"} Jul 31 23:38:34.924914 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:34.924867 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" event={"ID":"11037542-cc8e-4b76-9d41-90efd4829a01","Type":"ContainerStarted","Data":"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9"} Jul 31 23:38:34.942716 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:34.942666 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" podStartSLOduration=0.954553035 podStartE2EDuration="1.942649269s" totalImagesPullingTime="988.096234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:38:34.941008172 +0000 UTC m=+2502.863275956" watchObservedRunningTime="2026-07-31 23:38:34.942649269 +0000 UTC m=+2502.864917040" Jul 31 23:38:35.685156 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:35.685126 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:38:42.902230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:42.902190 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:38:43.695448 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:43.695411 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:43.695629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:43.695487 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:43.707699 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:43.707673 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:38:43.978529 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:38:43.978449 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:39:04.396152 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.396068 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:39:04.399311 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.399292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.401617 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.401594 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3c3302\"/\"kube-root-ca.crt\"" Jul 31 23:39:04.402707 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.402687 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3c3302\"/\"openshift-service-ca.crt\"" Jul 31 23:39:04.402865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.402847 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3c3302\"/\"default-dockercfg-bvdvv\"" Jul 31 23:39:04.402922 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.402899 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3c3302\"/\"test-model-3c3302-kserve-self-signed-certs\"" Jul 31 23:39:04.407062 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.407033 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:39:04.462713 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.462854 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.462854 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.462854 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.462995 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzr9s\" (UniqueName: \"kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.462995 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.462950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563360 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563504 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563504 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzr9s\" (UniqueName: \"kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563504 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563671 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563733 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563789 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563789 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563769 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.563931 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.563862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.565750 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.565727 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.566043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.566024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.570845 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.570800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzr9s\" (UniqueName: \"kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s\") pod \"test-model-3c3302-kserve-69cfbfcbc7-8vndr\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.709291 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.709267 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:04.841328 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:04.841297 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:39:04.843238 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:39:04.843210 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63c587f2_266d_41ef_b6ad_cd915c658c42.slice/crio-ee5934953413100fd74c4c30d14e2ddb29937cdad78d364e705604bf3c8f9906 WatchSource:0}: Error finding container ee5934953413100fd74c4c30d14e2ddb29937cdad78d364e705604bf3c8f9906: Status 404 returned error can't find the container with id ee5934953413100fd74c4c30d14e2ddb29937cdad78d364e705604bf3c8f9906 Jul 31 23:39:05.075132 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:05.075050 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" event={"ID":"63c587f2-266d-41ef-b6ad-cd915c658c42","Type":"ContainerStarted","Data":"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773"} Jul 31 23:39:05.075132 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:05.075095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" event={"ID":"63c587f2-266d-41ef-b6ad-cd915c658c42","Type":"ContainerStarted","Data":"ee5934953413100fd74c4c30d14e2ddb29937cdad78d364e705604bf3c8f9906"} Jul 31 23:39:05.094052 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:05.094005 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" podStartSLOduration=1.093990066 podStartE2EDuration="1.093990066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:39:05.09192216 +0000 UTC m=+2533.014189931" watchObservedRunningTime="2026-07-31 23:39:05.093990066 +0000 UTC m=+2533.016257838" Jul 31 23:39:07.490113 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:07.490076 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:39:14.710065 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:14.710030 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:14.710521 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:14.710121 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:14.722584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:14.722560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:15.138554 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:15.138488 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:39:15.496270 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:15.496236 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:39:44.407415 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:44.407381 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:39:45.308074 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.308042 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:39:45.435870 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.435835 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt"] Jul 31 23:39:45.438146 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.438129 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:45.441210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.441186 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:39:45.441325 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.441211 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:39:45.447166 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.447141 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt"] Jul 31 23:39:45.480187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.480165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt\" (UID: \"40bb075c-995e-4d7e-9a9f-d132df7d17b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:45.580566 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.580507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt\" (UID: \"40bb075c-995e-4d7e-9a9f-d132df7d17b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:45.581106 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.581088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt\" (UID: \"40bb075c-995e-4d7e-9a9f-d132df7d17b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:45.749659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.749629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:45.877530 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:45.877499 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt"] Jul 31 23:39:45.878946 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:39:45.878921 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb075c_995e_4d7e_9a9f_d132df7d17b1.slice/crio-34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28 WatchSource:0}: Error finding container 34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28: Status 404 returned error can't find the container with id 34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28 Jul 31 23:39:46.284323 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:46.284288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" event={"ID":"40bb075c-995e-4d7e-9a9f-d132df7d17b1","Type":"ContainerStarted","Data":"b739096b62947a430013a94cf9e3c2b03f16e0ccb69496e00af1020204ef4c19"} Jul 31 23:39:46.284464 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:46.284354 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" event={"ID":"40bb075c-995e-4d7e-9a9f-d132df7d17b1","Type":"ContainerStarted","Data":"34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28"} Jul 31 23:39:46.300313 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:46.300269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" podStartSLOduration=1.300255924 podStartE2EDuration="1.300255924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:39:46.29819414 +0000 UTC m=+2574.220461910" watchObservedRunningTime="2026-07-31 23:39:46.300255924 +0000 UTC m=+2574.222523673" Jul 31 23:39:47.290647 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:47.290611 2570 generic.go:356] "Generic (PLEG): container finished" podID="40bb075c-995e-4d7e-9a9f-d132df7d17b1" containerID="b739096b62947a430013a94cf9e3c2b03f16e0ccb69496e00af1020204ef4c19" exitCode=0 Jul 31 23:39:47.291107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:47.290672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" event={"ID":"40bb075c-995e-4d7e-9a9f-d132df7d17b1","Type":"ContainerDied","Data":"b739096b62947a430013a94cf9e3c2b03f16e0ccb69496e00af1020204ef4c19"} Jul 31 23:39:48.210115 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:48.210072 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:39:48.419856 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:48.419812 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:48.503486 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:48.503407 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca\") pod \"40bb075c-995e-4d7e-9a9f-d132df7d17b1\" (UID: \"40bb075c-995e-4d7e-9a9f-d132df7d17b1\") " Jul 31 23:39:48.503728 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:48.503704 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40bb075c-995e-4d7e-9a9f-d132df7d17b1" (UID: "40bb075c-995e-4d7e-9a9f-d132df7d17b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:39:48.604252 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:48.604222 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb075c-995e-4d7e-9a9f-d132df7d17b1-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:49.301273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:49.301239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" event={"ID":"40bb075c-995e-4d7e-9a9f-d132df7d17b1","Type":"ContainerDied","Data":"34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28"} Jul 31 23:39:49.301273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:49.301264 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt" Jul 31 23:39:49.301477 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:49.301286 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ad505ee5c28bc4c96d15c8cad1158574cfb40e4ba1c2f3a1b7ea8dac32fe28" Jul 31 23:39:56.261984 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:56.261946 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt"] Jul 31 23:39:56.264398 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:56.264375 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02002b-4w6mt"] Jul 31 23:39:56.538265 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:56.538190 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40bb075c-995e-4d7e-9a9f-d132df7d17b1" path="/var/lib/kubelet/pods/40bb075c-995e-4d7e-9a9f-d132df7d17b1/volumes" Jul 31 23:39:58.203920 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:58.203885 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:39:59.426063 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.425993 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" podUID="11037542-cc8e-4b76-9d41-90efd4829a01" containerName="main" containerID="cri-o://5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9" gracePeriod=45 Jul 31 23:39:59.670241 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.670218 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:39:59.791636 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791604 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp7wq\" (UniqueName: \"kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.791784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791679 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.791784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791703 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.791784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791724 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.791784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791747 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.791784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791774 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache\") pod \"11037542-cc8e-4b76-9d41-90efd4829a01\" (UID: \"11037542-cc8e-4b76-9d41-90efd4829a01\") " Jul 31 23:39:59.792078 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.791988 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home" (OuterVolumeSpecName: "home") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:39:59.792078 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.792020 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:39:59.792171 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.792083 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:59.792171 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.792090 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache" (OuterVolumeSpecName: "model-cache") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:39:59.793729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.793702 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:39:59.793729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.793716 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq" (OuterVolumeSpecName: "kube-api-access-kp7wq") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "kube-api-access-kp7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:39:59.793873 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.793706 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm" (OuterVolumeSpecName: "dshm") pod "11037542-cc8e-4b76-9d41-90efd4829a01" (UID: "11037542-cc8e-4b76-9d41-90efd4829a01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:39:59.893367 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.893341 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11037542-cc8e-4b76-9d41-90efd4829a01-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:59.893478 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.893364 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:59.893478 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.893389 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:59.893478 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.893398 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11037542-cc8e-4b76-9d41-90efd4829a01-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:39:59.893478 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:39:59.893407 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7wq\" (UniqueName: \"kubernetes.io/projected/11037542-cc8e-4b76-9d41-90efd4829a01-kube-api-access-kp7wq\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.326855 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.326774 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" podUID="63c587f2-266d-41ef-b6ad-cd915c658c42" containerName="main" containerID="cri-o://5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773" gracePeriod=45 Jul 31 23:40:00.359844 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.359796 2570 generic.go:356] "Generic (PLEG): container finished" podID="11037542-cc8e-4b76-9d41-90efd4829a01" containerID="5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9" exitCode=0 Jul 31 23:40:00.359947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.359853 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" event={"ID":"11037542-cc8e-4b76-9d41-90efd4829a01","Type":"ContainerDied","Data":"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9"} Jul 31 23:40:00.359947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.359875 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" Jul 31 23:40:00.359947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.359902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht" event={"ID":"11037542-cc8e-4b76-9d41-90efd4829a01","Type":"ContainerDied","Data":"f3b96afd88d7cbe51764f4fcc3e8ee8b1e0a67e4bd9efbc35649e9722ec0c862"} Jul 31 23:40:00.359947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.359920 2570 scope.go:117] "RemoveContainer" containerID="5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9" Jul 31 23:40:00.421642 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.421618 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:40:00.427722 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.427696 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-02002b/test-model-02002b-kserve-86f8df6745-zwjht"] Jul 31 23:40:00.443561 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.443544 2570 scope.go:117] "RemoveContainer" containerID="5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9" Jul 31 23:40:00.443939 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:40:00.443915 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9\": container with ID starting with 5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9 not found: ID does not exist" containerID="5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9" Jul 31 23:40:00.444002 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.443950 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9"} err="failed to get container status \"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9\": rpc error: code = NotFound desc = could not find container \"5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9\": container with ID starting with 5bf0e9955b7fab77b89f49753574c24733e4cac92b53c64083d354474a8a68f9 not found: ID does not exist" Jul 31 23:40:00.538810 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.538780 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11037542-cc8e-4b76-9d41-90efd4829a01" path="/var/lib/kubelet/pods/11037542-cc8e-4b76-9d41-90efd4829a01/volumes" Jul 31 23:40:00.573275 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.573253 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:40:00.700158 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700137 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700179 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzr9s\" (UniqueName: \"kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700225 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700245 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700267 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700284 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache\") pod \"63c587f2-266d-41ef-b6ad-cd915c658c42\" (UID: \"63c587f2-266d-41ef-b6ad-cd915c658c42\") " Jul 31 23:40:00.700559 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700528 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:40:00.700559 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700521 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home" (OuterVolumeSpecName: "home") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:40:00.700650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.700580 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache" (OuterVolumeSpecName: "model-cache") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:40:00.702373 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.702353 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:40:00.702473 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.702401 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm" (OuterVolumeSpecName: "dshm") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:40:00.702525 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.702466 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s" (OuterVolumeSpecName: "kube-api-access-zzr9s") pod "63c587f2-266d-41ef-b6ad-cd915c658c42" (UID: "63c587f2-266d-41ef-b6ad-cd915c658c42"). InnerVolumeSpecName "kube-api-access-zzr9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:40:00.801543 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801512 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzr9s\" (UniqueName: \"kubernetes.io/projected/63c587f2-266d-41ef-b6ad-cd915c658c42-kube-api-access-zzr9s\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.801543 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801540 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.801543 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801551 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.801757 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801559 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.801757 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801567 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63c587f2-266d-41ef-b6ad-cd915c658c42-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:00.801757 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:00.801575 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63c587f2-266d-41ef-b6ad-cd915c658c42-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:01.366272 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.366235 2570 generic.go:356] "Generic (PLEG): container finished" podID="63c587f2-266d-41ef-b6ad-cd915c658c42" containerID="5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773" exitCode=0 Jul 31 23:40:01.366425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.366280 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" Jul 31 23:40:01.366425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.366318 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" event={"ID":"63c587f2-266d-41ef-b6ad-cd915c658c42","Type":"ContainerDied","Data":"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773"} Jul 31 23:40:01.366425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.366358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr" event={"ID":"63c587f2-266d-41ef-b6ad-cd915c658c42","Type":"ContainerDied","Data":"ee5934953413100fd74c4c30d14e2ddb29937cdad78d364e705604bf3c8f9906"} Jul 31 23:40:01.366425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.366375 2570 scope.go:117] "RemoveContainer" containerID="5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773" Jul 31 23:40:01.375025 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.375007 2570 scope.go:117] "RemoveContainer" containerID="5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773" Jul 31 23:40:01.375266 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:40:01.375243 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773\": container with ID starting with 5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773 not found: ID does not exist" containerID="5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773" Jul 31 23:40:01.375312 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.375275 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773"} err="failed to get container status \"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773\": rpc error: code = NotFound desc = could not find container \"5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773\": container with ID starting with 5aea5d404eb1b0b10644e7dcaca2f25d9299f47734094792721db8ae077e2773 not found: ID does not exist" Jul 31 23:40:01.386635 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.386615 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:40:01.390906 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:01.390886 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3c3302/test-model-3c3302-kserve-69cfbfcbc7-8vndr"] Jul 31 23:40:02.538722 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:02.538688 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c587f2-266d-41ef-b6ad-cd915c658c42" path="/var/lib/kubelet/pods/63c587f2-266d-41ef-b6ad-cd915c658c42/volumes" Jul 31 23:40:03.060115 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060081 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz"] Jul 31 23:40:03.060393 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060381 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11037542-cc8e-4b76-9d41-90efd4829a01" containerName="main" Jul 31 23:40:03.060438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060395 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="11037542-cc8e-4b76-9d41-90efd4829a01" containerName="main" Jul 31 23:40:03.060438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060404 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c587f2-266d-41ef-b6ad-cd915c658c42" containerName="main" Jul 31 23:40:03.060438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060409 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c587f2-266d-41ef-b6ad-cd915c658c42" containerName="main" Jul 31 23:40:03.060438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060426 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40bb075c-995e-4d7e-9a9f-d132df7d17b1" containerName="revoke-keys" Jul 31 23:40:03.060438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060432 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="40bb075c-995e-4d7e-9a9f-d132df7d17b1" containerName="revoke-keys" Jul 31 23:40:03.060622 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060492 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="40bb075c-995e-4d7e-9a9f-d132df7d17b1" containerName="revoke-keys" Jul 31 23:40:03.060622 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060500 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c587f2-266d-41ef-b6ad-cd915c658c42" containerName="main" Jul 31 23:40:03.060622 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.060508 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="11037542-cc8e-4b76-9d41-90efd4829a01" containerName="main" Jul 31 23:40:03.063123 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.063105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:03.065588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.065565 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:40:03.065727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.065625 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:40:03.070480 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.070361 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz"] Jul 31 23:40:03.115630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.115600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz\" (UID: \"ea1c0508-b047-4d19-a789-0c53a8ded311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:03.216221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.216196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz\" (UID: \"ea1c0508-b047-4d19-a789-0c53a8ded311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:03.216778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.216760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz\" (UID: \"ea1c0508-b047-4d19-a789-0c53a8ded311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:03.374445 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.374388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:03.522456 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:40:03.522429 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1c0508_b047_4d19_a789_0c53a8ded311.slice/crio-0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f WatchSource:0}: Error finding container 0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f: Status 404 returned error can't find the container with id 0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f Jul 31 23:40:03.522771 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:03.522746 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz"] Jul 31 23:40:04.390810 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:04.390779 2570 generic.go:356] "Generic (PLEG): container finished" podID="ea1c0508-b047-4d19-a789-0c53a8ded311" containerID="81fd4ed7e5df89a18b6a0be9fcc7c464b22f621e609935bfecd96a81ab80b9b3" exitCode=0 Jul 31 23:40:04.391149 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:04.390863 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" event={"ID":"ea1c0508-b047-4d19-a789-0c53a8ded311","Type":"ContainerDied","Data":"81fd4ed7e5df89a18b6a0be9fcc7c464b22f621e609935bfecd96a81ab80b9b3"} Jul 31 23:40:04.391149 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:04.390915 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" event={"ID":"ea1c0508-b047-4d19-a789-0c53a8ded311","Type":"ContainerStarted","Data":"0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f"} Jul 31 23:40:05.520963 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:05.520938 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:05.634926 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:05.634895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca\") pod \"ea1c0508-b047-4d19-a789-0c53a8ded311\" (UID: \"ea1c0508-b047-4d19-a789-0c53a8ded311\") " Jul 31 23:40:05.635224 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:05.635203 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea1c0508-b047-4d19-a789-0c53a8ded311" (UID: "ea1c0508-b047-4d19-a789-0c53a8ded311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:40:05.735732 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:05.735708 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c0508-b047-4d19-a789-0c53a8ded311-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:40:06.403456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:06.403422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" event={"ID":"ea1c0508-b047-4d19-a789-0c53a8ded311","Type":"ContainerDied","Data":"0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f"} Jul 31 23:40:06.403626 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:06.403455 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz" Jul 31 23:40:06.403626 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:06.403470 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db5e040d92e02cd8170e2ddc962bf274f30b8aff6e7e0a6de50b165334de41f" Jul 31 23:40:13.688581 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:13.688546 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz"] Jul 31 23:40:13.695606 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:13.695580 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3c3302-wkthz"] Jul 31 23:40:14.538184 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:14.538152 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1c0508-b047-4d19-a789-0c53a8ded311" path="/var/lib/kubelet/pods/ea1c0508-b047-4d19-a789-0c53a8ded311/volumes" Jul 31 23:40:15.716389 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:15.716315 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:40:28.699877 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:28.699844 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:40:57.492714 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:40:57.492678 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:41:20.466648 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.466599 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj"] Jul 31 23:41:20.467047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.466949 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1c0508-b047-4d19-a789-0c53a8ded311" containerName="revoke-keys" Jul 31 23:41:20.467047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.466965 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1c0508-b047-4d19-a789-0c53a8ded311" containerName="revoke-keys" Jul 31 23:41:20.467047 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.467030 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1c0508-b047-4d19-a789-0c53a8ded311" containerName="revoke-keys" Jul 31 23:41:20.470014 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.469993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:20.472380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.472352 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:41:20.472483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.472352 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:41:20.476508 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.476481 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj"] Jul 31 23:41:20.568315 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.568284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj\" (UID: \"1a1f2f42-d561-4ff7-a29c-6ac46409d483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:20.669728 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.669688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj\" (UID: \"1a1f2f42-d561-4ff7-a29c-6ac46409d483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:20.670344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.670324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj\" (UID: \"1a1f2f42-d561-4ff7-a29c-6ac46409d483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:20.781209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.781131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:20.907914 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:20.907891 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj"] Jul 31 23:41:20.910124 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:41:20.910094 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1f2f42_d561_4ff7_a29c_6ac46409d483.slice/crio-836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214 WatchSource:0}: Error finding container 836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214: Status 404 returned error can't find the container with id 836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214 Jul 31 23:41:21.772754 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:21.772720 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a1f2f42-d561-4ff7-a29c-6ac46409d483" containerID="aa59c210d73743b10e117843b3c64f3e3b70974ffa2330bcd8276af18c593bd0" exitCode=0 Jul 31 23:41:21.773084 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:21.772805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" event={"ID":"1a1f2f42-d561-4ff7-a29c-6ac46409d483","Type":"ContainerDied","Data":"aa59c210d73743b10e117843b3c64f3e3b70974ffa2330bcd8276af18c593bd0"} Jul 31 23:41:21.773084 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:21.772906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" event={"ID":"1a1f2f42-d561-4ff7-a29c-6ac46409d483","Type":"ContainerStarted","Data":"836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214"} Jul 31 23:41:22.904013 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:22.903990 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:22.987106 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:22.987075 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca\") pod \"1a1f2f42-d561-4ff7-a29c-6ac46409d483\" (UID: \"1a1f2f42-d561-4ff7-a29c-6ac46409d483\") " Jul 31 23:41:22.987378 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:22.987355 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a1f2f42-d561-4ff7-a29c-6ac46409d483" (UID: "1a1f2f42-d561-4ff7-a29c-6ac46409d483"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:41:23.088323 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:23.088250 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a1f2f42-d561-4ff7-a29c-6ac46409d483-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:41:23.784620 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:23.784589 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" Jul 31 23:41:23.784785 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:23.784622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj" event={"ID":"1a1f2f42-d561-4ff7-a29c-6ac46409d483","Type":"ContainerDied","Data":"836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214"} Jul 31 23:41:23.784785 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:23.784667 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836991a58268eb81c7328aacdc38b6feb1f3b4b458219fa12fea40e6673b6214" Jul 31 23:41:36.139317 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:36.139278 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj"] Jul 31 23:41:36.141856 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:36.141835 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4734d6-rgwdj"] Jul 31 23:41:36.540601 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:36.540567 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1f2f42-d561-4ff7-a29c-6ac46409d483" path="/var/lib/kubelet/pods/1a1f2f42-d561-4ff7-a29c-6ac46409d483/volumes" Jul 31 23:41:38.294919 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:38.294879 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:41:42.708070 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.708037 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft"] Jul 31 23:41:42.708461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.708352 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1f2f42-d561-4ff7-a29c-6ac46409d483" containerName="revoke-keys" Jul 31 23:41:42.708461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.708363 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1f2f42-d561-4ff7-a29c-6ac46409d483" containerName="revoke-keys" Jul 31 23:41:42.708461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.708423 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1f2f42-d561-4ff7-a29c-6ac46409d483" containerName="revoke-keys" Jul 31 23:41:42.710613 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.710597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:42.713190 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.713168 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:41:42.713305 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.713179 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:41:42.717802 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.717778 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft"] Jul 31 23:41:42.827364 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.827340 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft\" (UID: \"6044daf8-38c2-4055-a930-2cbfcff3bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:42.927783 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.927753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft\" (UID: \"6044daf8-38c2-4055-a930-2cbfcff3bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:42.928346 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:42.928328 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft\" (UID: \"6044daf8-38c2-4055-a930-2cbfcff3bac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:43.022431 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:43.022377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:43.151874 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:43.151851 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft"] Jul 31 23:41:43.153049 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:41:43.153015 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6044daf8_38c2_4055_a930_2cbfcff3bac3.slice/crio-ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f WatchSource:0}: Error finding container ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f: Status 404 returned error can't find the container with id ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f Jul 31 23:41:43.886949 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:43.886919 2570 generic.go:356] "Generic (PLEG): container finished" podID="6044daf8-38c2-4055-a930-2cbfcff3bac3" containerID="5d339f50f54c2ffd76762e53948006acb6e83cafaef2c47e6cef98219e319663" exitCode=0 Jul 31 23:41:43.887282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:43.886969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" event={"ID":"6044daf8-38c2-4055-a930-2cbfcff3bac3","Type":"ContainerDied","Data":"5d339f50f54c2ffd76762e53948006acb6e83cafaef2c47e6cef98219e319663"} Jul 31 23:41:43.887282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:43.887022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" event={"ID":"6044daf8-38c2-4055-a930-2cbfcff3bac3","Type":"ContainerStarted","Data":"ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f"} Jul 31 23:41:45.019045 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.019015 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:45.142901 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.142773 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca\") pod \"6044daf8-38c2-4055-a930-2cbfcff3bac3\" (UID: \"6044daf8-38c2-4055-a930-2cbfcff3bac3\") " Jul 31 23:41:45.143107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.143084 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6044daf8-38c2-4055-a930-2cbfcff3bac3" (UID: "6044daf8-38c2-4055-a930-2cbfcff3bac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:41:45.244083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.244056 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6044daf8-38c2-4055-a930-2cbfcff3bac3-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:41:45.898728 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.898693 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" Jul 31 23:41:45.898926 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.898725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft" event={"ID":"6044daf8-38c2-4055-a930-2cbfcff3bac3","Type":"ContainerDied","Data":"ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f"} Jul 31 23:41:45.898926 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:45.898788 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab45eb5bab3d9721c34c8f123d5569e493de6a5bbdf2dc44ba35078294e14e0f" Jul 31 23:41:52.728658 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:52.728631 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:41:52.733222 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:52.733196 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:41:52.738458 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:52.738438 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:41:52.742278 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:52.742262 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:41:52.836972 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:52.836951 2570 scope.go:117] "RemoveContainer" containerID="55609de1113d27be456c9912894e7223e3499c4a0877126b0f26f6fc9098f369" Jul 31 23:41:58.367201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:58.367165 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft"] Jul 31 23:41:58.371320 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:58.371295 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7d39b-5wcft"] Jul 31 23:41:58.537666 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:41:58.537632 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6044daf8-38c2-4055-a930-2cbfcff3bac3" path="/var/lib/kubelet/pods/6044daf8-38c2-4055-a930-2cbfcff3bac3/volumes" Jul 31 23:42:00.196229 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:42:00.196193 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:42:12.584313 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:42:12.584277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:42:40.801442 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:42:40.801409 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:43:11.793310 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.793276 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:43:11.793777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.793621 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6044daf8-38c2-4055-a930-2cbfcff3bac3" containerName="revoke-keys" Jul 31 23:43:11.793777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.793633 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6044daf8-38c2-4055-a930-2cbfcff3bac3" containerName="revoke-keys" Jul 31 23:43:11.793777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.793700 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6044daf8-38c2-4055-a930-2cbfcff3bac3" containerName="revoke-keys" Jul 31 23:43:11.796601 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.796584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.801006 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.800542 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"kube-root-ca.crt\"" Jul 31 23:43:11.801006 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.800674 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"default-dockercfg-lsgv2\"" Jul 31 23:43:11.801006 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.800803 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"openshift-service-ca.crt\"" Jul 31 23:43:11.801006 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.800942 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"auth-test-model-99d0e4-kserve-self-signed-certs\"" Jul 31 23:43:11.803567 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.803540 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:43:11.867731 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867705 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.867881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.867881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rcg\" (UniqueName: \"kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.867881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.867881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.868071 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.867899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.968885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.968859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.968980 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.968907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.968980 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.968935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.968980 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.968959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rcg\" (UniqueName: \"kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.969086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.968989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.969086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.969018 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.969381 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.969362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.969440 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.969400 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.969482 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.969452 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.971177 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.971155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.971422 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.971408 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:11.976942 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:11.976921 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rcg\" (UniqueName: \"kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg\") pod \"auth-test-model-99d0e4-kserve-7d7b485675-zmnht\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:12.106862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:12.106785 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:12.233981 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:12.233956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:43:12.235650 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:43:12.235623 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf70d8623_7b74_4435_b6f9_81aeed6d1ccf.slice/crio-2c2990c562ca748d54609c1c4b9bc50ae1b847a519dae2876c05c14b8350affa WatchSource:0}: Error finding container 2c2990c562ca748d54609c1c4b9bc50ae1b847a519dae2876c05c14b8350affa: Status 404 returned error can't find the container with id 2c2990c562ca748d54609c1c4b9bc50ae1b847a519dae2876c05c14b8350affa Jul 31 23:43:12.341566 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:12.341534 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" event={"ID":"f70d8623-7b74-4435-b6f9-81aeed6d1ccf","Type":"ContainerStarted","Data":"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855"} Jul 31 23:43:12.341710 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:12.341580 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" event={"ID":"f70d8623-7b74-4435-b6f9-81aeed6d1ccf","Type":"ContainerStarted","Data":"2c2990c562ca748d54609c1c4b9bc50ae1b847a519dae2876c05c14b8350affa"} Jul 31 23:43:12.357838 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:12.357734 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" podStartSLOduration=1.3577193699999999 podStartE2EDuration="1.35771937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:43:12.357498136 +0000 UTC m=+2780.279765906" watchObservedRunningTime="2026-07-31 23:43:12.35771937 +0000 UTC m=+2780.279987135" Jul 31 23:43:14.485093 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:14.485058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:43:22.107003 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:22.106961 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:22.107003 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:22.107011 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:22.119301 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:22.119272 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:22.403107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:22.403024 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:43:41.792499 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.792458 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:43:41.795844 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.795803 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.799712 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.799684 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"openshift-service-ca.crt\"" Jul 31 23:43:41.799856 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.799736 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"kube-root-ca.crt\"" Jul 31 23:43:41.799856 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.799770 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"auth-test-model-3d60f7-kserve-self-signed-certs\"" Jul 31 23:43:41.799979 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.799863 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"default-dockercfg-4rv9s\"" Jul 31 23:43:41.805369 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.805343 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:43:41.879623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879599 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.879724 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.879724 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmkt\" (UniqueName: \"kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.879815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.879815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.879815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.879784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmkt\" (UniqueName: \"kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980738 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980738 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980738 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980927 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.980927 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.980771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.981146 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.981127 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.981205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.981148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.981244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.981211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.982807 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.982782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.983225 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.983203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:41.988687 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:41.988658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmkt\" (UniqueName: \"kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt\") pod \"auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:42.107151 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:42.107088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:42.237087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:42.237062 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:43:42.238918 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:43:42.238886 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9416e8ee_3bf2_48e4_a431_82d4af2d0be3.slice/crio-8864df1c10de9cd71894f2366e959385812c504437ebee2d6698ddf6af9be1b4 WatchSource:0}: Error finding container 8864df1c10de9cd71894f2366e959385812c504437ebee2d6698ddf6af9be1b4: Status 404 returned error can't find the container with id 8864df1c10de9cd71894f2366e959385812c504437ebee2d6698ddf6af9be1b4 Jul 31 23:43:42.495598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:42.495563 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" event={"ID":"9416e8ee-3bf2-48e4-a431-82d4af2d0be3","Type":"ContainerStarted","Data":"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682"} Jul 31 23:43:42.495752 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:42.495610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" event={"ID":"9416e8ee-3bf2-48e4-a431-82d4af2d0be3","Type":"ContainerStarted","Data":"8864df1c10de9cd71894f2366e959385812c504437ebee2d6698ddf6af9be1b4"} Jul 31 23:43:42.511695 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:42.511649 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" podStartSLOduration=1.511633944 podStartE2EDuration="1.511633944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:43:42.510677059 +0000 UTC m=+2810.432944831" watchObservedRunningTime="2026-07-31 23:43:42.511633944 +0000 UTC m=+2810.433901712" Jul 31 23:43:44.294963 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:44.294928 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:43:52.107246 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:52.107203 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:52.107653 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:52.107258 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:52.125116 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:52.125091 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:43:52.558120 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:43:52.558092 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:44:06.695922 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:44:06.695884 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:44:22.297305 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:44:22.297271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:44:33.707891 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:44:33.707855 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:44:48.497637 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:44:48.497555 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:45:00.150165 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.150130 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8"] Jul 31 23:45:00.153366 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.153349 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" Jul 31 23:45:00.157358 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.157333 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:45:00.157673 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.157655 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk"] Jul 31 23:45:00.161761 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.161741 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc"] Jul 31 23:45:00.161865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.161850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" Jul 31 23:45:00.164567 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.164543 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" Jul 31 23:45:00.165255 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.165227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" Jul 31 23:45:00.166972 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.166915 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8"] Jul 31 23:45:00.177069 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.176902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" Jul 31 23:45:00.194203 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.194173 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk"] Jul 31 23:45:00.197077 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.197038 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc"] Jul 31 23:45:00.215116 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.214735 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" Jul 31 23:45:00.324865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.324815 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8"] Jul 31 23:45:00.327666 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:45:00.327635 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2acd22_ee12_49d9_ab48_4c6dca92a863.slice/crio-291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95 WatchSource:0}: Error finding container 291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95: Status 404 returned error can't find the container with id 291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95 Jul 31 23:45:00.382497 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.382470 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc"] Jul 31 23:45:00.384190 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:45:00.384165 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba270735_14c4_4075_b65e_9a193b8c5784.slice/crio-bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27 WatchSource:0}: Error finding container bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27: Status 404 returned error can't find the container with id bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27 Jul 31 23:45:00.525968 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.525938 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:45:00.564081 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.564060 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk"] Jul 31 23:45:00.565450 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:45:00.565424 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3b97aa_9d5c_417f_99f3_4674adbc5f31.slice/crio-a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0 WatchSource:0}: Error finding container a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0: Status 404 returned error can't find the container with id a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0 Jul 31 23:45:00.905962 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.905929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" event={"ID":"ba270735-14c4-4075-b65e-9a193b8c5784","Type":"ContainerStarted","Data":"82c734bc8e184d5057af030370b3735ae58d0bfb6cd692ce877412410a688d9f"} Jul 31 23:45:00.906083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.905982 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" event={"ID":"ba270735-14c4-4075-b65e-9a193b8c5784","Type":"ContainerStarted","Data":"bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27"} Jul 31 23:45:00.907439 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.907415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" event={"ID":"5e3b97aa-9d5c-417f-99f3-4674adbc5f31","Type":"ContainerStarted","Data":"966e8382db9da232f893b02dc67fc5c69b4bee9fffa5a4ddefff7bd769d6c5ee"} Jul 31 23:45:00.907562 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.907463 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" event={"ID":"5e3b97aa-9d5c-417f-99f3-4674adbc5f31","Type":"ContainerStarted","Data":"a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0"} Jul 31 23:45:00.909010 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.908989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" event={"ID":"2b2acd22-ee12-49d9-ab48-4c6dca92a863","Type":"ContainerStarted","Data":"19d3c939692fc481124274860b37f40e35065cba2f9d047721b9bf9c149fee98"} Jul 31 23:45:00.909107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.909027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" event={"ID":"2b2acd22-ee12-49d9-ab48-4c6dca92a863","Type":"ContainerStarted","Data":"291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95"} Jul 31 23:45:00.925472 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.925431 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" podStartSLOduration=0.925419453 podStartE2EDuration="925.419453ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:45:00.923781282 +0000 UTC m=+2888.846049052" watchObservedRunningTime="2026-07-31 23:45:00.925419453 +0000 UTC m=+2888.847687217" Jul 31 23:45:00.976472 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.976432 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" podStartSLOduration=0.976420048 podStartE2EDuration="976.420048ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:45:00.973809607 +0000 UTC m=+2888.896077377" watchObservedRunningTime="2026-07-31 23:45:00.976420048 +0000 UTC m=+2888.898687819" Jul 31 23:45:00.976682 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:00.976662 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" podStartSLOduration=0.976655933 podStartE2EDuration="976.655933ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:45:00.945555168 +0000 UTC m=+2888.867822939" watchObservedRunningTime="2026-07-31 23:45:00.976655933 +0000 UTC m=+2888.898923682" Jul 31 23:45:01.919436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.919408 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e3b97aa-9d5c-417f-99f3-4674adbc5f31" containerID="966e8382db9da232f893b02dc67fc5c69b4bee9fffa5a4ddefff7bd769d6c5ee" exitCode=0 Jul 31 23:45:01.919894 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.919482 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" event={"ID":"5e3b97aa-9d5c-417f-99f3-4674adbc5f31","Type":"ContainerDied","Data":"966e8382db9da232f893b02dc67fc5c69b4bee9fffa5a4ddefff7bd769d6c5ee"} Jul 31 23:45:01.921045 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.921024 2570 generic.go:356] "Generic (PLEG): container finished" podID="2b2acd22-ee12-49d9-ab48-4c6dca92a863" containerID="19d3c939692fc481124274860b37f40e35065cba2f9d047721b9bf9c149fee98" exitCode=0 Jul 31 23:45:01.921161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.921073 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" event={"ID":"2b2acd22-ee12-49d9-ab48-4c6dca92a863","Type":"ContainerDied","Data":"19d3c939692fc481124274860b37f40e35065cba2f9d047721b9bf9c149fee98"} Jul 31 23:45:01.922576 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.922550 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba270735-14c4-4075-b65e-9a193b8c5784" containerID="82c734bc8e184d5057af030370b3735ae58d0bfb6cd692ce877412410a688d9f" exitCode=0 Jul 31 23:45:01.922659 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:01.922589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" event={"ID":"ba270735-14c4-4075-b65e-9a193b8c5784","Type":"ContainerDied","Data":"82c734bc8e184d5057af030370b3735ae58d0bfb6cd692ce877412410a688d9f"} Jul 31 23:45:03.069141 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.069113 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" Jul 31 23:45:03.117778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.117758 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" Jul 31 23:45:03.120655 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.120633 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" Jul 31 23:45:03.934039 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.934005 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" Jul 31 23:45:03.934221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.934041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc" event={"ID":"ba270735-14c4-4075-b65e-9a193b8c5784","Type":"ContainerDied","Data":"bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27"} Jul 31 23:45:03.934221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.934081 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc6f0d5c087f4f2fcc26d3f14eac2af5307f75d8d7e3ff729ee09dc0dd95cb27" Jul 31 23:45:03.935333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.935311 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" Jul 31 23:45:03.935333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.935316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759025-v8zqk" event={"ID":"5e3b97aa-9d5c-417f-99f3-4674adbc5f31","Type":"ContainerDied","Data":"a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0"} Jul 31 23:45:03.935538 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.935357 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e7f88def4e08bc1a27ae296279942d47e46fba114c65fe146a0b5c52d52fd0" Jul 31 23:45:03.936614 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.936596 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" event={"ID":"2b2acd22-ee12-49d9-ab48-4c6dca92a863","Type":"ContainerDied","Data":"291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95"} Jul 31 23:45:03.936697 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.936614 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8" Jul 31 23:45:03.936697 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:03.936627 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291c556a463fb3b0a5068b2949e394176e7cb83f220d3959c1fa175d97b96c95" Jul 31 23:45:14.800149 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:14.800114 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:45:26.617512 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:26.617476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:45:41.389726 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:41.389689 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:45:52.878438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:52.878404 2570 scope.go:117] "RemoveContainer" containerID="b739096b62947a430013a94cf9e3c2b03f16e0ccb69496e00af1020204ef4c19" Jul 31 23:45:53.092474 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:45:53.092444 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:46:07.600020 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:07.599984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:46:19.498708 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:19.498619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:46:33.997333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:33.997296 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:46:37.840641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:37.840608 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:46:38.608991 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:38.608959 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:46:43.993340 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:43.993295 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:46:52.761318 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.761291 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:46:52.765302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.765280 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:46:52.772383 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.772364 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:46:52.776144 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.776125 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:46:52.860119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.860090 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" podUID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" containerName="main" containerID="cri-o://d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855" gracePeriod=45 Jul 31 23:46:52.897938 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:52.897913 2570 scope.go:117] "RemoveContainer" containerID="81fd4ed7e5df89a18b6a0be9fcc7c464b22f621e609935bfecd96a81ab80b9b3" Jul 31 23:46:53.105260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.105232 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:46:53.177589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177559 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.177765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177622 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.177765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177646 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26rcg\" (UniqueName: \"kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.177765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177698 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.177765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177732 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.177765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177765 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache\") pod \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\" (UID: \"f70d8623-7b74-4435-b6f9-81aeed6d1ccf\") " Jul 31 23:46:53.178068 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.177878 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home" (OuterVolumeSpecName: "home") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.178068 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.178018 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.178150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.178062 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.178150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.178085 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.179767 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.179747 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm" (OuterVolumeSpecName: "dshm") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.179864 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.179815 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:46:53.179989 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.179895 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg" (OuterVolumeSpecName: "kube-api-access-26rcg") pod "f70d8623-7b74-4435-b6f9-81aeed6d1ccf" (UID: "f70d8623-7b74-4435-b6f9-81aeed6d1ccf"). InnerVolumeSpecName "kube-api-access-26rcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:46:53.279023 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.278998 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.279023 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.279019 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.279150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.279030 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.279150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.279042 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rcg\" (UniqueName: \"kubernetes.io/projected/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-kube-api-access-26rcg\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.279150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.279052 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70d8623-7b74-4435-b6f9-81aeed6d1ccf-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.516230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.516198 2570 generic.go:356] "Generic (PLEG): container finished" podID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" containerID="d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855" exitCode=0 Jul 31 23:46:53.516380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.516256 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" Jul 31 23:46:53.516380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.516282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" event={"ID":"f70d8623-7b74-4435-b6f9-81aeed6d1ccf","Type":"ContainerDied","Data":"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855"} Jul 31 23:46:53.516380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.516327 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht" event={"ID":"f70d8623-7b74-4435-b6f9-81aeed6d1ccf","Type":"ContainerDied","Data":"2c2990c562ca748d54609c1c4b9bc50ae1b847a519dae2876c05c14b8350affa"} Jul 31 23:46:53.516380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.516351 2570 scope.go:117] "RemoveContainer" containerID="d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855" Jul 31 23:46:53.525398 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.525381 2570 scope.go:117] "RemoveContainer" containerID="d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855" Jul 31 23:46:53.525642 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:46:53.525624 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855\": container with ID starting with d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855 not found: ID does not exist" containerID="d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855" Jul 31 23:46:53.525697 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.525651 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855"} err="failed to get container status \"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855\": rpc error: code = NotFound desc = could not find container \"d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855\": container with ID starting with d18d9835537412c74e011d55f81154f8b0bf9d2b6086b5c90c469b37c89fe855 not found: ID does not exist" Jul 31 23:46:53.537414 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.537391 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:46:53.541023 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.541005 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/auth-test-model-99d0e4-kserve-7d7b485675-zmnht"] Jul 31 23:46:53.627669 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.627641 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" podUID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" containerName="main" containerID="cri-o://81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682" gracePeriod=45 Jul 31 23:46:53.861204 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.861181 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:46:53.983597 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983568 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983617 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983641 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983684 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983924 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983730 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nmkt\" (UniqueName: \"kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983924 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983759 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm\") pod \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\" (UID: \"9416e8ee-3bf2-48e4-a431-82d4af2d0be3\") " Jul 31 23:46:53.983924 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983886 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.984086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983918 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home" (OuterVolumeSpecName: "home") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.984086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.983967 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.984086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.984045 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.984086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.984062 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.984086 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.984076 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:53.985993 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.985970 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm" (OuterVolumeSpecName: "dshm") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:46:53.985993 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.985979 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:46:53.986116 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:53.986018 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt" (OuterVolumeSpecName: "kube-api-access-5nmkt") pod "9416e8ee-3bf2-48e4-a431-82d4af2d0be3" (UID: "9416e8ee-3bf2-48e4-a431-82d4af2d0be3"). InnerVolumeSpecName "kube-api-access-5nmkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:46:54.084623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.084546 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:54.084623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.084576 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nmkt\" (UniqueName: \"kubernetes.io/projected/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-kube-api-access-5nmkt\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:54.084623 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.084587 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9416e8ee-3bf2-48e4-a431-82d4af2d0be3-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:46:54.524201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.524160 2570 generic.go:356] "Generic (PLEG): container finished" podID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" containerID="81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682" exitCode=0 Jul 31 23:46:54.524201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.524197 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" Jul 31 23:46:54.524403 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.524242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" event={"ID":"9416e8ee-3bf2-48e4-a431-82d4af2d0be3","Type":"ContainerDied","Data":"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682"} Jul 31 23:46:54.524403 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.524277 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd" event={"ID":"9416e8ee-3bf2-48e4-a431-82d4af2d0be3","Type":"ContainerDied","Data":"8864df1c10de9cd71894f2366e959385812c504437ebee2d6698ddf6af9be1b4"} Jul 31 23:46:54.524403 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.524298 2570 scope.go:117] "RemoveContainer" containerID="81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682" Jul 31 23:46:54.533750 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.533727 2570 scope.go:117] "RemoveContainer" containerID="81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682" Jul 31 23:46:54.534012 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:46:54.533991 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682\": container with ID starting with 81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682 not found: ID does not exist" containerID="81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682" Jul 31 23:46:54.534072 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.534019 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682"} err="failed to get container status \"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682\": rpc error: code = NotFound desc = could not find container \"81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682\": container with ID starting with 81d405f708ba922944ec2242325bdfa1ad93149a3fa6c9514b0bc82caeb1c682 not found: ID does not exist" Jul 31 23:46:54.538866 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.538843 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" path="/var/lib/kubelet/pods/f70d8623-7b74-4435-b6f9-81aeed6d1ccf/volumes" Jul 31 23:46:54.544670 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.544652 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:46:54.548784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:54.548759 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/auth-test-model-3d60f7-kserve-775c4bc8b7-nm8wd"] Jul 31 23:46:56.537708 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:56.537675 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" path="/var/lib/kubelet/pods/9416e8ee-3bf2-48e4-a431-82d4af2d0be3/volumes" Jul 31 23:46:58.594664 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594630 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594937 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594948 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594963 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2acd22-ee12-49d9-ab48-4c6dca92a863" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594969 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2acd22-ee12-49d9-ab48-4c6dca92a863" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594976 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba270735-14c4-4075-b65e-9a193b8c5784" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594981 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba270735-14c4-4075-b65e-9a193b8c5784" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594987 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3b97aa-9d5c-417f-99f3-4674adbc5f31" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594992 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3b97aa-9d5c-417f-99f3-4674adbc5f31" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.594999 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595004 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595073 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70d8623-7b74-4435-b6f9-81aeed6d1ccf" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595082 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3b97aa-9d5c-417f-99f3-4674adbc5f31" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595089 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9416e8ee-3bf2-48e4-a431-82d4af2d0be3" containerName="main" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595094 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2acd22-ee12-49d9-ab48-4c6dca92a863" containerName="cleanup" Jul 31 23:46:58.595119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.595101 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba270735-14c4-4075-b65e-9a193b8c5784" containerName="cleanup" Jul 31 23:46:58.599107 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.599089 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.602539 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.602513 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"openshift-service-ca.crt\"" Jul 31 23:46:58.602640 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.602534 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"sub-test-model-99d0e4-kserve-self-signed-certs\"" Jul 31 23:46:58.602640 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.602521 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"default-dockercfg-lsgv2\"" Jul 31 23:46:58.602640 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.602513 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"kube-root-ca.crt\"" Jul 31 23:46:58.608354 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.608330 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:46:58.719691 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.719691 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719695 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.719935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.719935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.719935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vn5l\" (UniqueName: \"kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.719935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.719916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820529 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vn5l\" (UniqueName: \"kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820529 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.820778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.821034 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.820985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.821034 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.821016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.821120 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.821064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.822772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.822753 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.823154 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.823134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.828197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.828174 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vn5l\" (UniqueName: \"kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l\") pod \"sub-test-model-99d0e4-kserve-88776cf49-59q72\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:58.909236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:58.909150 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:46:59.038994 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:59.038956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:46:59.039868 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:46:59.039811 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dcf3f7b_5ede_4a1a_a27c_d3e2da331873.slice/crio-f992b2151aedc558ec2262ef9aa1dcb21c1eabb331d3ccf0e519e980165fb84e WatchSource:0}: Error finding container f992b2151aedc558ec2262ef9aa1dcb21c1eabb331d3ccf0e519e980165fb84e: Status 404 returned error can't find the container with id f992b2151aedc558ec2262ef9aa1dcb21c1eabb331d3ccf0e519e980165fb84e Jul 31 23:46:59.551718 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:59.551683 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" event={"ID":"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873","Type":"ContainerStarted","Data":"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669"} Jul 31 23:46:59.551913 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:59.551725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" event={"ID":"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873","Type":"ContainerStarted","Data":"f992b2151aedc558ec2262ef9aa1dcb21c1eabb331d3ccf0e519e980165fb84e"} Jul 31 23:46:59.570221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:46:59.570171 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" podStartSLOduration=1.5701561769999999 podStartE2EDuration="1.570156177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:59.567514538 +0000 UTC m=+3007.489782309" watchObservedRunningTime="2026-07-31 23:46:59.570156177 +0000 UTC m=+3007.492424347" Jul 31 23:47:00.992799 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:00.992766 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:47:08.909694 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:08.909654 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:47:08.910140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:08.909713 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:47:08.922403 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:08.922372 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:47:09.611253 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:09.611222 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:47:17.987514 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.987477 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:47:17.991375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.991351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:17.994888 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.994867 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"default-dockercfg-4rv9s\"" Jul 31 23:47:17.995111 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.994871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"openshift-service-ca.crt\"" Jul 31 23:47:17.995111 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.994946 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"sub-test-model-3d60f7-kserve-self-signed-certs\"" Jul 31 23:47:17.995283 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:17.994971 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"kube-root-ca.crt\"" Jul 31 23:47:18.003187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.003165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:47:18.076947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.076923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.077087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.076951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.077087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.077003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.077087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.077069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.077251 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.077113 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.077251 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.077187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pd8l\" (UniqueName: \"kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178444 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pd8l\" (UniqueName: \"kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.178989 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.179055 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.178970 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.179055 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.179027 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.180701 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.180684 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.180887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.180870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.186244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.186218 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pd8l\" (UniqueName: \"kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l\") pod \"sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.302109 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.302020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:18.432263 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.432234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:47:18.433740 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:47:18.433713 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c85efa6_10d2_4548_9d65_e6a520a46f94.slice/crio-d04478368b9cbfc07138db02da8752588f36d1c66dc0848913a7e66a7e43622c WatchSource:0}: Error finding container d04478368b9cbfc07138db02da8752588f36d1c66dc0848913a7e66a7e43622c: Status 404 returned error can't find the container with id d04478368b9cbfc07138db02da8752588f36d1c66dc0848913a7e66a7e43622c Jul 31 23:47:18.646548 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.646452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" event={"ID":"6c85efa6-10d2-4548-9d65-e6a520a46f94","Type":"ContainerStarted","Data":"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662"} Jul 31 23:47:18.646548 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.646503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" event={"ID":"6c85efa6-10d2-4548-9d65-e6a520a46f94","Type":"ContainerStarted","Data":"d04478368b9cbfc07138db02da8752588f36d1c66dc0848913a7e66a7e43622c"} Jul 31 23:47:18.662448 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:18.662404 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" podStartSLOduration=1.662388944 podStartE2EDuration="1.662388944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.661897864 +0000 UTC m=+3026.584165633" watchObservedRunningTime="2026-07-31 23:47:18.662388944 +0000 UTC m=+3026.584656714" Jul 31 23:47:20.496082 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:20.496051 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:47:28.302881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:28.302840 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:28.303380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:28.302898 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:28.315704 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:28.315674 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:28.714755 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:28.714728 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:47:52.984866 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:52.984810 2570 scope.go:117] "RemoveContainer" containerID="aa59c210d73743b10e117843b3c64f3e3b70974ffa2330bcd8276af18c593bd0" Jul 31 23:47:52.993360 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:52.993343 2570 scope.go:117] "RemoveContainer" containerID="5d339f50f54c2ffd76762e53948006acb6e83cafaef2c47e6cef98219e319663" Jul 31 23:47:56.211310 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:47:56.211277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:48:08.698376 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:08.698339 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:48:29.501748 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:29.501714 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:48:34.074180 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:34.074145 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:48:34.880939 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:34.880904 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:48:40.089670 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.089623 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:48:40.590173 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.590144 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:48:40.593036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.593020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.597629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.597610 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99d0e4\"/\"rate-test-model-99d0e4-kserve-self-signed-certs\"" Jul 31 23:48:40.604172 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.604151 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:48:40.706729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706705 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.706902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fq9\" (UniqueName: \"kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.706902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.706902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706878 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.707043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.707043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.706936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.807636 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.807772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.807772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.807772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.807772 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fq9\" (UniqueName: \"kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.808070 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.807778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.808150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.808126 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.808199 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.808155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.808236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.808213 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.809944 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.809917 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.810247 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.810224 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.816336 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.816314 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fq9\" (UniqueName: \"kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9\") pod \"rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:40.902022 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:40.901943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:41.031130 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:41.029165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:48:41.034038 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:48:41.034008 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7320d4cd_8246_4120_b17e_09abe062ac57.slice/crio-277d2bd26b7b4fd9b4153e413b57da7783e2221ded9e53f7de4ce261b7c58063 WatchSource:0}: Error finding container 277d2bd26b7b4fd9b4153e413b57da7783e2221ded9e53f7de4ce261b7c58063: Status 404 returned error can't find the container with id 277d2bd26b7b4fd9b4153e413b57da7783e2221ded9e53f7de4ce261b7c58063 Jul 31 23:48:41.095042 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:41.095010 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" event={"ID":"7320d4cd-8246-4120-b17e-09abe062ac57","Type":"ContainerStarted","Data":"155afc6a45d9ba6269f3c620bab631a4f735618d487eb3f2603fd5aecf22484f"} Jul 31 23:48:41.095373 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:41.095060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" event={"ID":"7320d4cd-8246-4120-b17e-09abe062ac57","Type":"ContainerStarted","Data":"277d2bd26b7b4fd9b4153e413b57da7783e2221ded9e53f7de4ce261b7c58063"} Jul 31 23:48:41.112715 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:41.112674 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" podStartSLOduration=1.112660104 podStartE2EDuration="1.112660104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:41.110126034 +0000 UTC m=+3109.032393808" watchObservedRunningTime="2026-07-31 23:48:41.112660104 +0000 UTC m=+3109.034927875" Jul 31 23:48:46.086068 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:46.086032 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:48:49.092447 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.092386 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" podUID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" containerName="main" containerID="cri-o://3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669" gracePeriod=45 Jul 31 23:48:49.347400 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.347346 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:48:49.475018 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.474984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475038 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475074 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475093 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475209 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475111 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vn5l\" (UniqueName: \"kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l\") pod \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\" (UID: \"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873\") " Jul 31 23:48:49.475483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475361 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:49.475483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475389 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:49.475483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.475401 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home" (OuterVolumeSpecName: "home") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:49.477059 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.477034 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm" (OuterVolumeSpecName: "dshm") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:49.477195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.477077 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:48:49.477195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.477156 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l" (OuterVolumeSpecName: "kube-api-access-4vn5l") pod "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" (UID: "8dcf3f7b-5ede-4a1a-a27c-d3e2da331873"). InnerVolumeSpecName "kube-api-access-4vn5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:48:49.576458 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576426 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.576598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576472 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.576598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576482 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.576598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576490 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.576598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576498 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vn5l\" (UniqueName: \"kubernetes.io/projected/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-kube-api-access-4vn5l\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.576598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.576508 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:49.898936 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:49.898904 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" podUID="6c85efa6-10d2-4548-9d65-e6a520a46f94" containerName="main" containerID="cri-o://fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662" gracePeriod=45 Jul 31 23:48:50.142082 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.142060 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:48:50.145508 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.145484 2570 generic.go:356] "Generic (PLEG): container finished" podID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" containerID="3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669" exitCode=0 Jul 31 23:48:50.145624 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.145527 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" Jul 31 23:48:50.145624 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.145569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" event={"ID":"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873","Type":"ContainerDied","Data":"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669"} Jul 31 23:48:50.145624 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.145608 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72" event={"ID":"8dcf3f7b-5ede-4a1a-a27c-d3e2da331873","Type":"ContainerDied","Data":"f992b2151aedc558ec2262ef9aa1dcb21c1eabb331d3ccf0e519e980165fb84e"} Jul 31 23:48:50.145790 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.145627 2570 scope.go:117] "RemoveContainer" containerID="3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669" Jul 31 23:48:50.147028 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.147003 2570 generic.go:356] "Generic (PLEG): container finished" podID="6c85efa6-10d2-4548-9d65-e6a520a46f94" containerID="fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662" exitCode=0 Jul 31 23:48:50.147152 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.147032 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" event={"ID":"6c85efa6-10d2-4548-9d65-e6a520a46f94","Type":"ContainerDied","Data":"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662"} Jul 31 23:48:50.147152 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.147049 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" Jul 31 23:48:50.147152 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.147060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8" event={"ID":"6c85efa6-10d2-4548-9d65-e6a520a46f94","Type":"ContainerDied","Data":"d04478368b9cbfc07138db02da8752588f36d1c66dc0848913a7e66a7e43622c"} Jul 31 23:48:50.155054 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.155038 2570 scope.go:117] "RemoveContainer" containerID="3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669" Jul 31 23:48:50.155340 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:48:50.155317 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669\": container with ID starting with 3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669 not found: ID does not exist" containerID="3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669" Jul 31 23:48:50.155424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.155348 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669"} err="failed to get container status \"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669\": rpc error: code = NotFound desc = could not find container \"3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669\": container with ID starting with 3e357acdacf01b8eff278fbc949aa7ee861adb373efa8fb6c670d1129b952669 not found: ID does not exist" Jul 31 23:48:50.155424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.155365 2570 scope.go:117] "RemoveContainer" containerID="fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662" Jul 31 23:48:50.164844 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.164808 2570 scope.go:117] "RemoveContainer" containerID="fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662" Jul 31 23:48:50.165136 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:48:50.165117 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662\": container with ID starting with fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662 not found: ID does not exist" containerID="fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662" Jul 31 23:48:50.165215 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.165141 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662"} err="failed to get container status \"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662\": rpc error: code = NotFound desc = could not find container \"fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662\": container with ID starting with fcc4f8bd94505f3537feafb204cdacffd551f251b6a905c516b60e68ba787662 not found: ID does not exist" Jul 31 23:48:50.172190 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.172166 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:48:50.175390 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.175369 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/sub-test-model-99d0e4-kserve-88776cf49-59q72"] Jul 31 23:48:50.280564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280537 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280660 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280591 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280660 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pd8l\" (UniqueName: \"kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280660 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280635 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280660 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280658 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280841 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280677 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs\") pod \"6c85efa6-10d2-4548-9d65-e6a520a46f94\" (UID: \"6c85efa6-10d2-4548-9d65-e6a520a46f94\") " Jul 31 23:48:50.280894 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280816 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:50.280973 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280934 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:50.280973 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.280950 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home" (OuterVolumeSpecName: "home") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:50.281082 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.281029 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.281082 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.281044 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.281082 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.281053 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.282792 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.282766 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm" (OuterVolumeSpecName: "dshm") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:48:50.282905 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.282886 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:48:50.282960 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.282903 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l" (OuterVolumeSpecName: "kube-api-access-5pd8l") pod "6c85efa6-10d2-4548-9d65-e6a520a46f94" (UID: "6c85efa6-10d2-4548-9d65-e6a520a46f94"). InnerVolumeSpecName "kube-api-access-5pd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:48:50.381861 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.381808 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pd8l\" (UniqueName: \"kubernetes.io/projected/6c85efa6-10d2-4548-9d65-e6a520a46f94-kube-api-access-5pd8l\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.381861 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.381856 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85efa6-10d2-4548-9d65-e6a520a46f94-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.382036 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.381869 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85efa6-10d2-4548-9d65-e6a520a46f94-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:48:50.468587 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.468558 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:48:50.472053 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.472030 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/sub-test-model-3d60f7-kserve-7bbdffb699-4qvq8"] Jul 31 23:48:50.539344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.539316 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c85efa6-10d2-4548-9d65-e6a520a46f94" path="/var/lib/kubelet/pods/6c85efa6-10d2-4548-9d65-e6a520a46f94/volumes" Jul 31 23:48:50.539708 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.539694 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" path="/var/lib/kubelet/pods/8dcf3f7b-5ede-4a1a-a27c-d3e2da331873/volumes" Jul 31 23:48:50.902875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.902777 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:50.902875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.902837 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:50.915151 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:50.915127 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:48:51.165204 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:48:51.165128 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:49:05.891767 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.891732 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:49:05.892344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892197 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" containerName="main" Jul 31 23:49:05.892344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892217 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" containerName="main" Jul 31 23:49:05.892344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892253 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c85efa6-10d2-4548-9d65-e6a520a46f94" containerName="main" Jul 31 23:49:05.892344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892261 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c85efa6-10d2-4548-9d65-e6a520a46f94" containerName="main" Jul 31 23:49:05.892344 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892345 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c85efa6-10d2-4548-9d65-e6a520a46f94" containerName="main" Jul 31 23:49:05.892608 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.892359 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcf3f7b-5ede-4a1a-a27c-d3e2da331873" containerName="main" Jul 31 23:49:05.895392 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.895372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.902259 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.902239 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"rate-test-model-3d60f7-kserve-self-signed-certs\"" Jul 31 23:49:05.903256 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.903239 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"default-dockercfg-4rv9s\"" Jul 31 23:49:05.903357 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.903241 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"kube-root-ca.crt\"" Jul 31 23:49:05.904574 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.904557 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3d60f7\"/\"openshift-service-ca.crt\"" Jul 31 23:49:05.929023 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.929002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:49:05.998490 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.998598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.998598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.998598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hvl\" (UniqueName: \"kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.998598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:05.998801 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:05.998686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099120 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hvl\" (UniqueName: \"kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099244 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099416 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099416 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099530 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099473 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099530 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.099687 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.099565 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.107743 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.107713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.107880 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.107764 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.108529 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.108508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hvl\" (UniqueName: \"kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl\") pod \"rate-test-model-3d60f7-kserve-ffc595b75-sggk2\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.203814 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.203792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:06.334729 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:06.334701 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:49:06.336520 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:49:06.336492 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6342e51_d901_43f8_a997_3f6f39d25b62.slice/crio-1584bf07a7adce78211ad6bc5a4f916b1a8361fa9de32f1161ae6ca1a15e192f WatchSource:0}: Error finding container 1584bf07a7adce78211ad6bc5a4f916b1a8361fa9de32f1161ae6ca1a15e192f: Status 404 returned error can't find the container with id 1584bf07a7adce78211ad6bc5a4f916b1a8361fa9de32f1161ae6ca1a15e192f Jul 31 23:49:07.239058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:07.239015 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" event={"ID":"d6342e51-d901-43f8-a997-3f6f39d25b62","Type":"ContainerStarted","Data":"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276"} Jul 31 23:49:07.239058 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:07.239063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" event={"ID":"d6342e51-d901-43f8-a997-3f6f39d25b62","Type":"ContainerStarted","Data":"1584bf07a7adce78211ad6bc5a4f916b1a8361fa9de32f1161ae6ca1a15e192f"} Jul 31 23:49:07.257319 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:07.257269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" podStartSLOduration=2.257255034 podStartE2EDuration="2.257255034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:49:07.25512259 +0000 UTC m=+3135.177390361" watchObservedRunningTime="2026-07-31 23:49:07.257255034 +0000 UTC m=+3135.179522806" Jul 31 23:49:08.595009 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:08.594975 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:49:16.204927 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:16.204846 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:16.204927 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:16.204909 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:16.216998 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:16.216968 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:16.294197 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:16.294171 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:49:46.995524 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:46.995490 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:49:55.087585 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:49:55.087548 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:50:11.102218 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:11.102181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:50:16.483607 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:16.483573 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:50:17.318273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:17.318240 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:50:18.225815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.225778 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716232368"] Jul 31 23:50:18.229054 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.229037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:18.231486 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.231465 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9vjlt\"" Jul 31 23:50:18.236091 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.236068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716232368"] Jul 31 23:50:18.335804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.335775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gh6l\" (UniqueName: \"kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l\") pod \"test-curl-7373-140026716232368\" (UID: \"49403ac3-0e00-42b6-b351-d56dec4639a6\") " pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:18.436417 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.436391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gh6l\" (UniqueName: \"kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l\") pod \"test-curl-7373-140026716232368\" (UID: \"49403ac3-0e00-42b6-b351-d56dec4639a6\") " pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:18.444615 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.444592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gh6l\" (UniqueName: \"kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l\") pod \"test-curl-7373-140026716232368\" (UID: \"49403ac3-0e00-42b6-b351-d56dec4639a6\") " pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:18.539055 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.538993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:18.665908 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.665883 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716232368"] Jul 31 23:50:18.668183 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:18.668155 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49403ac3_0e00_42b6_b351_d56dec4639a6.slice/crio-bcc929036d1652869a34a9d57ad0a9f604b46454dedd8e6f549961f305fd7da0 WatchSource:0}: Error finding container bcc929036d1652869a34a9d57ad0a9f604b46454dedd8e6f549961f305fd7da0: Status 404 returned error can't find the container with id bcc929036d1652869a34a9d57ad0a9f604b46454dedd8e6f549961f305fd7da0 Jul 31 23:50:18.669560 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:18.669541 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:50:19.606504 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:19.606427 2570 generic.go:356] "Generic (PLEG): container finished" podID="49403ac3-0e00-42b6-b351-d56dec4639a6" containerID="05734b0ade0be24109fbf9fac7a32ea493ebb9f5b347d38370796e320f73c368" exitCode=0 Jul 31 23:50:19.606871 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:19.606517 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716232368" event={"ID":"49403ac3-0e00-42b6-b351-d56dec4639a6","Type":"ContainerDied","Data":"05734b0ade0be24109fbf9fac7a32ea493ebb9f5b347d38370796e320f73c368"} Jul 31 23:50:19.606871 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:19.606566 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716232368" event={"ID":"49403ac3-0e00-42b6-b351-d56dec4639a6","Type":"ContainerStarted","Data":"bcc929036d1652869a34a9d57ad0a9f604b46454dedd8e6f549961f305fd7da0"} Jul 31 23:50:19.996097 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:19.996062 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:50:20.739831 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.739796 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:20.775387 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.775362 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716232368_49403ac3-0e00-42b6-b351-d56dec4639a6/test-curl-7373-140026716232368/0.log" Jul 31 23:50:20.796784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.796745 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716232368"] Jul 31 23:50:20.803125 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.803085 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716232368"] Jul 31 23:50:20.857053 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.857028 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gh6l\" (UniqueName: \"kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l\") pod \"49403ac3-0e00-42b6-b351-d56dec4639a6\" (UID: \"49403ac3-0e00-42b6-b351-d56dec4639a6\") " Jul 31 23:50:20.859068 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.859046 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l" (OuterVolumeSpecName: "kube-api-access-9gh6l") pod "49403ac3-0e00-42b6-b351-d56dec4639a6" (UID: "49403ac3-0e00-42b6-b351-d56dec4639a6"). InnerVolumeSpecName "kube-api-access-9gh6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:20.913437 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.913405 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716287280"] Jul 31 23:50:20.913791 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.913772 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" containerName="test-curl-7373-140026716232368" Jul 31 23:50:20.913917 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.913794 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" containerName="test-curl-7373-140026716232368" Jul 31 23:50:20.913995 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.913923 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" containerName="test-curl-7373-140026716232368" Jul 31 23:50:20.916875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.916848 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:20.923378 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.923357 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716287280"] Jul 31 23:50:20.957866 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:20.957840 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gh6l\" (UniqueName: \"kubernetes.io/projected/49403ac3-0e00-42b6-b351-d56dec4639a6-kube-api-access-9gh6l\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:21.058564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.058539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbzn\" (UniqueName: \"kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn\") pod \"test-curl-7373-140026716287280\" (UID: \"885324aa-a468-4cfb-a212-839ef93fa7e6\") " pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:21.159634 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.159566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbzn\" (UniqueName: \"kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn\") pod \"test-curl-7373-140026716287280\" (UID: \"885324aa-a468-4cfb-a212-839ef93fa7e6\") " pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:21.168205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.168183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbzn\" (UniqueName: \"kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn\") pod \"test-curl-7373-140026716287280\" (UID: \"885324aa-a468-4cfb-a212-839ef93fa7e6\") " pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:21.226153 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.226131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:21.351226 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.351200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716287280"] Jul 31 23:50:21.352646 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:21.352618 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod885324aa_a468_4cfb_a212_839ef93fa7e6.slice/crio-f9a03f4ccdb0fbb72e2349a92fdca26f6b69d893246fa0bc879b2978cc28b314 WatchSource:0}: Error finding container f9a03f4ccdb0fbb72e2349a92fdca26f6b69d893246fa0bc879b2978cc28b314: Status 404 returned error can't find the container with id f9a03f4ccdb0fbb72e2349a92fdca26f6b69d893246fa0bc879b2978cc28b314 Jul 31 23:50:21.617624 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.617586 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:21.617746 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.617585 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc929036d1652869a34a9d57ad0a9f604b46454dedd8e6f549961f305fd7da0" Jul 31 23:50:21.618865 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:21.618837 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716287280" event={"ID":"885324aa-a468-4cfb-a212-839ef93fa7e6","Type":"ContainerStarted","Data":"f9a03f4ccdb0fbb72e2349a92fdca26f6b69d893246fa0bc879b2978cc28b314"} Jul 31 23:50:21.629493 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:50:21.629463 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7373-140026716232368\" is forbidden: User \"system:node:ip-10-0-137-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-58.ec2.internal' and this object" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:22.537674 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:50:22.537632 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7373-140026716232368\" is forbidden: User \"system:node:ip-10-0-137-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-58.ec2.internal' and this object" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" pod="opendatahub/test-curl-7373-140026716232368" Jul 31 23:50:22.538610 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:22.538586 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49403ac3-0e00-42b6-b351-d56dec4639a6" path="/var/lib/kubelet/pods/49403ac3-0e00-42b6-b351-d56dec4639a6/volumes" Jul 31 23:50:22.625238 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:22.625208 2570 generic.go:356] "Generic (PLEG): container finished" podID="885324aa-a468-4cfb-a212-839ef93fa7e6" containerID="2041c67cd5a00f818babe6e1f6e8e2da2c0b192c0e75af63fac47ed31162fc99" exitCode=0 Jul 31 23:50:22.625387 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:22.625287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716287280" event={"ID":"885324aa-a468-4cfb-a212-839ef93fa7e6","Type":"ContainerDied","Data":"2041c67cd5a00f818babe6e1f6e8e2da2c0b192c0e75af63fac47ed31162fc99"} Jul 31 23:50:23.761475 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.761452 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:23.780440 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.780415 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716287280_885324aa-a468-4cfb-a212-839ef93fa7e6/test-curl-7373-140026716287280/0.log" Jul 31 23:50:23.802113 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.802080 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716287280"] Jul 31 23:50:23.805315 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.805291 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716287280"] Jul 31 23:50:23.882481 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.882423 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dbzn\" (UniqueName: \"kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn\") pod \"885324aa-a468-4cfb-a212-839ef93fa7e6\" (UID: \"885324aa-a468-4cfb-a212-839ef93fa7e6\") " Jul 31 23:50:23.884577 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.884544 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn" (OuterVolumeSpecName: "kube-api-access-8dbzn") pod "885324aa-a468-4cfb-a212-839ef93fa7e6" (UID: "885324aa-a468-4cfb-a212-839ef93fa7e6"). InnerVolumeSpecName "kube-api-access-8dbzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:23.966455 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.966422 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:23.966804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.966787 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="885324aa-a468-4cfb-a212-839ef93fa7e6" containerName="test-curl-7373-140026716287280" Jul 31 23:50:23.966916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.966807 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="885324aa-a468-4cfb-a212-839ef93fa7e6" containerName="test-curl-7373-140026716287280" Jul 31 23:50:23.966916 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.966913 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="885324aa-a468-4cfb-a212-839ef93fa7e6" containerName="test-curl-7373-140026716287280" Jul 31 23:50:23.969774 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.969756 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:23.977207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.977187 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:23.983223 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:23.983195 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbzn\" (UniqueName: \"kubernetes.io/projected/885324aa-a468-4cfb-a212-839ef93fa7e6-kube-api-access-8dbzn\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:24.084180 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.084157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b299b\" (UniqueName: \"kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b\") pod \"test-curl-7373-140026716331296\" (UID: \"413af013-3c85-4da7-b57b-a3d2c90e42ea\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:24.185387 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.185331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b299b\" (UniqueName: \"kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b\") pod \"test-curl-7373-140026716331296\" (UID: \"413af013-3c85-4da7-b57b-a3d2c90e42ea\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:24.193500 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.193480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b299b\" (UniqueName: \"kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b\") pod \"test-curl-7373-140026716331296\" (UID: \"413af013-3c85-4da7-b57b-a3d2c90e42ea\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:24.279638 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.279619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:24.404326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.404299 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:24.405784 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:24.405753 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod413af013_3c85_4da7_b57b_a3d2c90e42ea.slice/crio-0e4a1b3d9210ff9969a21ddbaf05b3f44adef872400e75f67058961f6c2673af WatchSource:0}: Error finding container 0e4a1b3d9210ff9969a21ddbaf05b3f44adef872400e75f67058961f6c2673af: Status 404 returned error can't find the container with id 0e4a1b3d9210ff9969a21ddbaf05b3f44adef872400e75f67058961f6c2673af Jul 31 23:50:24.545290 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.545259 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="885324aa-a468-4cfb-a212-839ef93fa7e6" path="/var/lib/kubelet/pods/885324aa-a468-4cfb-a212-839ef93fa7e6/volumes" Jul 31 23:50:24.636319 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.636296 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716287280" Jul 31 23:50:24.636436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.636337 2570 scope.go:117] "RemoveContainer" containerID="2041c67cd5a00f818babe6e1f6e8e2da2c0b192c0e75af63fac47ed31162fc99" Jul 31 23:50:24.637637 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:24.637559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716331296" event={"ID":"413af013-3c85-4da7-b57b-a3d2c90e42ea","Type":"ContainerStarted","Data":"0e4a1b3d9210ff9969a21ddbaf05b3f44adef872400e75f67058961f6c2673af"} Jul 31 23:50:25.646554 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:25.646520 2570 generic.go:356] "Generic (PLEG): container finished" podID="413af013-3c85-4da7-b57b-a3d2c90e42ea" containerID="8cd62c3a603a8d9c7ff159af8bbc44c5e744f038fa6d67fed1e1a852c874fc76" exitCode=0 Jul 31 23:50:25.646949 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:25.646569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716331296" event={"ID":"413af013-3c85-4da7-b57b-a3d2c90e42ea","Type":"ContainerDied","Data":"8cd62c3a603a8d9c7ff159af8bbc44c5e744f038fa6d67fed1e1a852c874fc76"} Jul 31 23:50:26.780519 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.780496 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:26.799099 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.799080 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716331296_413af013-3c85-4da7-b57b-a3d2c90e42ea/test-curl-7373-140026716331296/0.log" Jul 31 23:50:26.805788 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.805767 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b299b\" (UniqueName: \"kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b\") pod \"413af013-3c85-4da7-b57b-a3d2c90e42ea\" (UID: \"413af013-3c85-4da7-b57b-a3d2c90e42ea\") " Jul 31 23:50:26.807741 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.807721 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b" (OuterVolumeSpecName: "kube-api-access-b299b") pod "413af013-3c85-4da7-b57b-a3d2c90e42ea" (UID: "413af013-3c85-4da7-b57b-a3d2c90e42ea"). InnerVolumeSpecName "kube-api-access-b299b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:26.824159 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.824132 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:26.829463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.829445 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:26.906902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.906853 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b299b\" (UniqueName: \"kubernetes.io/projected/413af013-3c85-4da7-b57b-a3d2c90e42ea-kube-api-access-b299b\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:26.950399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.950376 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:26.950668 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.950656 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413af013-3c85-4da7-b57b-a3d2c90e42ea" containerName="test-curl-7373-140026716331296" Jul 31 23:50:26.950720 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.950669 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="413af013-3c85-4da7-b57b-a3d2c90e42ea" containerName="test-curl-7373-140026716331296" Jul 31 23:50:26.950753 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.950739 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="413af013-3c85-4da7-b57b-a3d2c90e42ea" containerName="test-curl-7373-140026716331296" Jul 31 23:50:26.953709 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.953696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:26.959954 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:26.959935 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:27.007836 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.007791 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks54m\" (UniqueName: \"kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m\") pod \"test-curl-7373-140026716331296\" (UID: \"a9b30130-0f40-4a9a-9e22-c29b9ccebc26\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:27.108686 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.108658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks54m\" (UniqueName: \"kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m\") pod \"test-curl-7373-140026716331296\" (UID: \"a9b30130-0f40-4a9a-9e22-c29b9ccebc26\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:27.117104 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.117083 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks54m\" (UniqueName: \"kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m\") pod \"test-curl-7373-140026716331296\" (UID: \"a9b30130-0f40-4a9a-9e22-c29b9ccebc26\") " pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:27.263758 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.263734 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:27.388894 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.388868 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:27.390407 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:27.390375 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b30130_0f40_4a9a_9e22_c29b9ccebc26.slice/crio-4f7022f4b46655e6fd3d8e8107ecd8be0104510bc538a39b517760237ee3ac77 WatchSource:0}: Error finding container 4f7022f4b46655e6fd3d8e8107ecd8be0104510bc538a39b517760237ee3ac77: Status 404 returned error can't find the container with id 4f7022f4b46655e6fd3d8e8107ecd8be0104510bc538a39b517760237ee3ac77 Jul 31 23:50:27.658210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.658146 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e4a1b3d9210ff9969a21ddbaf05b3f44adef872400e75f67058961f6c2673af" Jul 31 23:50:27.658210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.658147 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:27.659342 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.659314 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716331296" event={"ID":"a9b30130-0f40-4a9a-9e22-c29b9ccebc26","Type":"ContainerStarted","Data":"4f7022f4b46655e6fd3d8e8107ecd8be0104510bc538a39b517760237ee3ac77"} Jul 31 23:50:27.669556 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:27.669526 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7373-140026716331296" oldPodUID="413af013-3c85-4da7-b57b-a3d2c90e42ea" podUID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" Jul 31 23:50:28.538876 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:28.538816 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413af013-3c85-4da7-b57b-a3d2c90e42ea" path="/var/lib/kubelet/pods/413af013-3c85-4da7-b57b-a3d2c90e42ea/volumes" Jul 31 23:50:28.665204 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:28.665175 2570 generic.go:356] "Generic (PLEG): container finished" podID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" containerID="bf52f3d5d0b027a2688d989afe04b5c638e4d2f446b9527e4cab232f84073655" exitCode=0 Jul 31 23:50:28.665356 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:28.665261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716331296" event={"ID":"a9b30130-0f40-4a9a-9e22-c29b9ccebc26","Type":"ContainerDied","Data":"bf52f3d5d0b027a2688d989afe04b5c638e4d2f446b9527e4cab232f84073655"} Jul 31 23:50:29.797922 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.797897 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:29.817347 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.817310 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716331296_a9b30130-0f40-4a9a-9e22-c29b9ccebc26/test-curl-7373-140026716331296/0.log" Jul 31 23:50:29.829699 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.829672 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks54m\" (UniqueName: \"kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m\") pod \"a9b30130-0f40-4a9a-9e22-c29b9ccebc26\" (UID: \"a9b30130-0f40-4a9a-9e22-c29b9ccebc26\") " Jul 31 23:50:29.831769 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.831739 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m" (OuterVolumeSpecName: "kube-api-access-ks54m") pod "a9b30130-0f40-4a9a-9e22-c29b9ccebc26" (UID: "a9b30130-0f40-4a9a-9e22-c29b9ccebc26"). InnerVolumeSpecName "kube-api-access-ks54m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:29.845591 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.845568 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:29.850739 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.850720 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716331296"] Jul 31 23:50:29.931119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.931063 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks54m\" (UniqueName: \"kubernetes.io/projected/a9b30130-0f40-4a9a-9e22-c29b9ccebc26-kube-api-access-ks54m\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:29.968315 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.968293 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:29.968622 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.968610 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" containerName="test-curl-7373-140026716331296" Jul 31 23:50:29.968662 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.968624 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" containerName="test-curl-7373-140026716331296" Jul 31 23:50:29.968706 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.968678 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" containerName="test-curl-7373-140026716331296" Jul 31 23:50:29.971450 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.971434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:29.980383 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:29.980365 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:30.031707 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.031678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49px\" (UniqueName: \"kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px\") pod \"test-curl-7373-140026716414288\" (UID: \"2cd7db10-9943-4380-9194-f198f2ab5391\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:30.132432 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.132409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49px\" (UniqueName: \"kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px\") pod \"test-curl-7373-140026716414288\" (UID: \"2cd7db10-9943-4380-9194-f198f2ab5391\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:30.142544 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.142521 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49px\" (UniqueName: \"kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px\") pod \"test-curl-7373-140026716414288\" (UID: \"2cd7db10-9943-4380-9194-f198f2ab5391\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:30.281288 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.281265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:30.411907 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.411880 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:30.413476 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:30.413444 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd7db10_9943_4380_9194_f198f2ab5391.slice/crio-47bf4b2ab286a712c924be2425ce850a0825b3f298831b0f4e444d38f3727c88 WatchSource:0}: Error finding container 47bf4b2ab286a712c924be2425ce850a0825b3f298831b0f4e444d38f3727c88: Status 404 returned error can't find the container with id 47bf4b2ab286a712c924be2425ce850a0825b3f298831b0f4e444d38f3727c88 Jul 31 23:50:30.539185 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.539121 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b30130-0f40-4a9a-9e22-c29b9ccebc26" path="/var/lib/kubelet/pods/a9b30130-0f40-4a9a-9e22-c29b9ccebc26/volumes" Jul 31 23:50:30.677198 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.677173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716414288" event={"ID":"2cd7db10-9943-4380-9194-f198f2ab5391","Type":"ContainerStarted","Data":"47bf4b2ab286a712c924be2425ce850a0825b3f298831b0f4e444d38f3727c88"} Jul 31 23:50:30.678486 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.678466 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716331296" Jul 31 23:50:30.678486 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:30.678480 2570 scope.go:117] "RemoveContainer" containerID="bf52f3d5d0b027a2688d989afe04b5c638e4d2f446b9527e4cab232f84073655" Jul 31 23:50:31.502059 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.501999 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" podUID="7320d4cd-8246-4120-b17e-09abe062ac57" containerName="main" containerID="cri-o://155afc6a45d9ba6269f3c620bab631a4f735618d487eb3f2603fd5aecf22484f" gracePeriod=45 Jul 31 23:50:31.686105 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.686077 2570 generic.go:356] "Generic (PLEG): container finished" podID="7320d4cd-8246-4120-b17e-09abe062ac57" containerID="155afc6a45d9ba6269f3c620bab631a4f735618d487eb3f2603fd5aecf22484f" exitCode=0 Jul 31 23:50:31.686254 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.686161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" event={"ID":"7320d4cd-8246-4120-b17e-09abe062ac57","Type":"ContainerDied","Data":"155afc6a45d9ba6269f3c620bab631a4f735618d487eb3f2603fd5aecf22484f"} Jul 31 23:50:31.687939 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.687914 2570 generic.go:356] "Generic (PLEG): container finished" podID="2cd7db10-9943-4380-9194-f198f2ab5391" containerID="9ecb2d275b0e0783058d6ac5064d48b0431479ae30c93d77a542d121052c9a30" exitCode=0 Jul 31 23:50:31.688044 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.687996 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716414288" event={"ID":"2cd7db10-9943-4380-9194-f198f2ab5391","Type":"ContainerDied","Data":"9ecb2d275b0e0783058d6ac5064d48b0431479ae30c93d77a542d121052c9a30"} Jul 31 23:50:31.756628 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.756574 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:50:31.846404 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846378 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846436 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98fq9\" (UniqueName: \"kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846478 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846494 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846511 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846542 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs\") pod \"7320d4cd-8246-4120-b17e-09abe062ac57\" (UID: \"7320d4cd-8246-4120-b17e-09abe062ac57\") " Jul 31 23:50:31.846815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846771 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache" (OuterVolumeSpecName: "model-cache") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:31.846815 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846786 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:31.846925 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.846800 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home" (OuterVolumeSpecName: "home") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:31.848492 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.848460 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:50:31.848581 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.848511 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm" (OuterVolumeSpecName: "dshm") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:31.848630 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.848569 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9" (OuterVolumeSpecName: "kube-api-access-98fq9") pod "7320d4cd-8246-4120-b17e-09abe062ac57" (UID: "7320d4cd-8246-4120-b17e-09abe062ac57"). InnerVolumeSpecName "kube-api-access-98fq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:31.947778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947754 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:31.947778 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947776 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98fq9\" (UniqueName: \"kubernetes.io/projected/7320d4cd-8246-4120-b17e-09abe062ac57-kube-api-access-98fq9\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:31.947966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947787 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:31.947966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947797 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:31.947966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947805 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7320d4cd-8246-4120-b17e-09abe062ac57-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:31.947966 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:31.947812 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7320d4cd-8246-4120-b17e-09abe062ac57-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.336781 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.336746 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" podUID="d6342e51-d901-43f8-a997-3f6f39d25b62" containerName="main" containerID="cri-o://e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276" gracePeriod=45 Jul 31 23:50:32.588242 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.588184 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:50:32.652809 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652782 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.652976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652837 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.652976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652869 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.652976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652896 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27hvl\" (UniqueName: \"kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.652976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652918 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.652976 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.652966 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir\") pod \"d6342e51-d901-43f8-a997-3f6f39d25b62\" (UID: \"d6342e51-d901-43f8-a997-3f6f39d25b62\") " Jul 31 23:50:32.653232 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.653151 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home" (OuterVolumeSpecName: "home") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:32.653232 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.653197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:32.653349 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.653305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:32.654806 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.654784 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm" (OuterVolumeSpecName: "dshm") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:50:32.654935 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.654853 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 23:50:32.655009 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.654993 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl" (OuterVolumeSpecName: "kube-api-access-27hvl") pod "d6342e51-d901-43f8-a997-3f6f39d25b62" (UID: "d6342e51-d901-43f8-a997-3f6f39d25b62"). InnerVolumeSpecName "kube-api-access-27hvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:32.697887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.697870 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" Jul 31 23:50:32.697997 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.697902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8" event={"ID":"7320d4cd-8246-4120-b17e-09abe062ac57","Type":"ContainerDied","Data":"277d2bd26b7b4fd9b4153e413b57da7783e2221ded9e53f7de4ce261b7c58063"} Jul 31 23:50:32.697997 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.697956 2570 scope.go:117] "RemoveContainer" containerID="155afc6a45d9ba6269f3c620bab631a4f735618d487eb3f2603fd5aecf22484f" Jul 31 23:50:32.699452 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.699425 2570 generic.go:356] "Generic (PLEG): container finished" podID="d6342e51-d901-43f8-a997-3f6f39d25b62" containerID="e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276" exitCode=0 Jul 31 23:50:32.699564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.699455 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" Jul 31 23:50:32.699564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.699519 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" event={"ID":"d6342e51-d901-43f8-a997-3f6f39d25b62","Type":"ContainerDied","Data":"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276"} Jul 31 23:50:32.699564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.699549 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2" event={"ID":"d6342e51-d901-43f8-a997-3f6f39d25b62","Type":"ContainerDied","Data":"1584bf07a7adce78211ad6bc5a4f916b1a8361fa9de32f1161ae6ca1a15e192f"} Jul 31 23:50:32.708362 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.708346 2570 scope.go:117] "RemoveContainer" containerID="e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276" Jul 31 23:50:32.716097 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.716077 2570 scope.go:117] "RemoveContainer" containerID="e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276" Jul 31 23:50:32.716407 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:50:32.716378 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276\": container with ID starting with e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276 not found: ID does not exist" containerID="e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276" Jul 31 23:50:32.716485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.716431 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276"} err="failed to get container status \"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276\": rpc error: code = NotFound desc = could not find container \"e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276\": container with ID starting with e9322b853bc316d25bff71c8619e4aecc5ac064a4234909f34ef2488aff94276 not found: ID does not exist" Jul 31 23:50:32.722077 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.722047 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:50:32.728352 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.728331 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99d0e4/rate-test-model-99d0e4-kserve-6c854d68d7-2b5t8"] Jul 31 23:50:32.739688 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.739668 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:50:32.743057 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.743032 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3d60f7/rate-test-model-3d60f7-kserve-ffc595b75-sggk2"] Jul 31 23:50:32.754380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754360 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hvl\" (UniqueName: \"kubernetes.io/projected/d6342e51-d901-43f8-a997-3f6f39d25b62-kube-api-access-27hvl\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.754444 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754389 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-model-cache\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.754444 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754431 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-tmp-dir\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.754515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754446 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-dshm\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.754515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754460 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6342e51-d901-43f8-a997-3f6f39d25b62-tls-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.754515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.754473 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6342e51-d901-43f8-a997-3f6f39d25b62-home\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.814146 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.814122 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:32.833219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.833189 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716414288_2cd7db10-9943-4380-9194-f198f2ab5391/test-curl-7373-140026716414288/0.log" Jul 31 23:50:32.855111 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.855084 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:32.855333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.855296 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n49px\" (UniqueName: \"kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px\") pod \"2cd7db10-9943-4380-9194-f198f2ab5391\" (UID: \"2cd7db10-9943-4380-9194-f198f2ab5391\") " Jul 31 23:50:32.857480 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.857457 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px" (OuterVolumeSpecName: "kube-api-access-n49px") pod "2cd7db10-9943-4380-9194-f198f2ab5391" (UID: "2cd7db10-9943-4380-9194-f198f2ab5391"). InnerVolumeSpecName "kube-api-access-n49px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:32.858507 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.858487 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:32.956232 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.956208 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49px\" (UniqueName: \"kubernetes.io/projected/2cd7db10-9943-4380-9194-f198f2ab5391-kube-api-access-n49px\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:32.971302 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971280 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:32.971580 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971569 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6342e51-d901-43f8-a997-3f6f39d25b62" containerName="main" Jul 31 23:50:32.971629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971582 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6342e51-d901-43f8-a997-3f6f39d25b62" containerName="main" Jul 31 23:50:32.971629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971598 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7320d4cd-8246-4120-b17e-09abe062ac57" containerName="main" Jul 31 23:50:32.971629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971603 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7320d4cd-8246-4120-b17e-09abe062ac57" containerName="main" Jul 31 23:50:32.971629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971619 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd7db10-9943-4380-9194-f198f2ab5391" containerName="test-curl-7373-140026716414288" Jul 31 23:50:32.971629 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971625 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd7db10-9943-4380-9194-f198f2ab5391" containerName="test-curl-7373-140026716414288" Jul 31 23:50:32.971780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971674 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7320d4cd-8246-4120-b17e-09abe062ac57" containerName="main" Jul 31 23:50:32.971780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971684 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd7db10-9943-4380-9194-f198f2ab5391" containerName="test-curl-7373-140026716414288" Jul 31 23:50:32.971780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.971691 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6342e51-d901-43f8-a997-3f6f39d25b62" containerName="main" Jul 31 23:50:32.974437 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.974422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:32.981315 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:32.981276 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:33.057027 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.057001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65s2\" (UniqueName: \"kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2\") pod \"test-curl-7373-140026716414288\" (UID: \"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:33.157967 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.157902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65s2\" (UniqueName: \"kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2\") pod \"test-curl-7373-140026716414288\" (UID: \"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:33.166687 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.166657 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65s2\" (UniqueName: \"kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2\") pod \"test-curl-7373-140026716414288\" (UID: \"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc\") " pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:33.284120 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.284099 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:33.412092 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.412067 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:33.413503 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:33.413477 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9deaa625_ad1b_4c8f_ac62_acc8934c6dbc.slice/crio-e53037832e8f9340431c909a362974b9b8c9f5cfc63001d78bf0350fa1634154 WatchSource:0}: Error finding container e53037832e8f9340431c909a362974b9b8c9f5cfc63001d78bf0350fa1634154: Status 404 returned error can't find the container with id e53037832e8f9340431c909a362974b9b8c9f5cfc63001d78bf0350fa1634154 Jul 31 23:50:33.707043 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.707023 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:33.707375 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.707049 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47bf4b2ab286a712c924be2425ce850a0825b3f298831b0f4e444d38f3727c88" Jul 31 23:50:33.709338 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.709306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716414288" event={"ID":"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc","Type":"ContainerStarted","Data":"e53037832e8f9340431c909a362974b9b8c9f5cfc63001d78bf0350fa1634154"} Jul 31 23:50:33.728948 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:33.728926 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7373-140026716414288" oldPodUID="2cd7db10-9943-4380-9194-f198f2ab5391" podUID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" Jul 31 23:50:34.538451 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:34.538415 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd7db10-9943-4380-9194-f198f2ab5391" path="/var/lib/kubelet/pods/2cd7db10-9943-4380-9194-f198f2ab5391/volumes" Jul 31 23:50:34.538799 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:34.538785 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7320d4cd-8246-4120-b17e-09abe062ac57" path="/var/lib/kubelet/pods/7320d4cd-8246-4120-b17e-09abe062ac57/volumes" Jul 31 23:50:34.539157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:34.539144 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6342e51-d901-43f8-a997-3f6f39d25b62" path="/var/lib/kubelet/pods/d6342e51-d901-43f8-a997-3f6f39d25b62/volumes" Jul 31 23:50:34.715684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:34.715653 2570 generic.go:356] "Generic (PLEG): container finished" podID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" containerID="d65e9a9bc3b17752228f6f8cc722c5348fe9fd8323331a0d5f3d0474b1dec392" exitCode=0 Jul 31 23:50:34.716020 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:34.715715 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7373-140026716414288" event={"ID":"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc","Type":"ContainerDied","Data":"d65e9a9bc3b17752228f6f8cc722c5348fe9fd8323331a0d5f3d0474b1dec392"} Jul 31 23:50:35.855682 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.855652 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:35.874858 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.874816 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7373-140026716414288_9deaa625-ad1b-4c8f-ac62-acc8934c6dbc/test-curl-7373-140026716414288/0.log" Jul 31 23:50:35.875899 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.875877 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b65s2\" (UniqueName: \"kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2\") pod \"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc\" (UID: \"9deaa625-ad1b-4c8f-ac62-acc8934c6dbc\") " Jul 31 23:50:35.877841 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.877804 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2" (OuterVolumeSpecName: "kube-api-access-b65s2") pod "9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" (UID: "9deaa625-ad1b-4c8f-ac62-acc8934c6dbc"). InnerVolumeSpecName "kube-api-access-b65s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:50:35.896424 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.896399 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:35.899899 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.899877 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7373-140026716414288"] Jul 31 23:50:35.977124 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:35.977059 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b65s2\" (UniqueName: \"kubernetes.io/projected/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc-kube-api-access-b65s2\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:50:36.538040 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.538001 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" path="/var/lib/kubelet/pods/9deaa625-ad1b-4c8f-ac62-acc8934c6dbc/volumes" Jul 31 23:50:36.629425 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.629393 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:50:36.629851 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.629813 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" containerName="test-curl-7373-140026716414288" Jul 31 23:50:36.629851 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.629852 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" containerName="test-curl-7373-140026716414288" Jul 31 23:50:36.630083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.629963 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deaa625-ad1b-4c8f-ac62-acc8934c6dbc" containerName="test-curl-7373-140026716414288" Jul 31 23:50:36.632910 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.632891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.635873 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.635851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c16217-openshift-default-dockercfg-tqtdm\"" Jul 31 23:50:36.635970 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.635886 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 31 23:50:36.646095 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.646072 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:50:36.682170 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682143 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682271 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682271 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682271 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xs5\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682399 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.682515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.682407 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.726737 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.726712 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7373-140026716414288" Jul 31 23:50:36.726737 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.726729 2570 scope.go:117] "RemoveContainer" containerID="d65e9a9bc3b17752228f6f8cc722c5348fe9fd8323331a0d5f3d0474b1dec392" Jul 31 23:50:36.783764 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.783734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xs5\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.783887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.783769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.783887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.783812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.783887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.783865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784056 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784121 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784174 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784174 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784278 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784278 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784238 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784388 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784341 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784437 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784611 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.784740 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.784723 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.786195 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.786175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.786380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.786358 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.791411 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.791363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.791476 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.791460 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xs5\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5\") pod \"e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:36.944679 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:36.944648 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:37.077201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:37.077177 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:50:37.079088 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:50:37.079056 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2528601_3938_4ff0_9534_4068f56fc6ee.slice/crio-b828a283d940832dc534748c4ede934b336b7d4c9b3c991078ee28790d6baa85 WatchSource:0}: Error finding container b828a283d940832dc534748c4ede934b336b7d4c9b3c991078ee28790d6baa85: Status 404 returned error can't find the container with id b828a283d940832dc534748c4ede934b336b7d4c9b3c991078ee28790d6baa85 Jul 31 23:50:37.734273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:37.734238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerStarted","Data":"b828a283d940832dc534748c4ede934b336b7d4c9b3c991078ee28790d6baa85"} Jul 31 23:50:40.239365 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.239331 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:40.239625 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.239403 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:40.239625 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.239435 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:40.754654 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.754618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerStarted","Data":"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3"} Jul 31 23:50:40.776101 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.776054 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podStartSLOduration=1.618365217 podStartE2EDuration="4.776038842s" totalImagesPullingTime="3.157673625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:40.773283385 +0000 UTC m=+3228.695551157" watchObservedRunningTime="2026-07-31 23:50:40.776038842 +0000 UTC m=+3228.698306612" Jul 31 23:50:40.945084 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:40.945051 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:41.950144 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:41.950115 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:42.765893 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:42.765860 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:42.766898 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:42.766875 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:46.987734 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:46.987660 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:50:57.767583 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.767546 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:50:57.768032 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.767616 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:50:57.846315 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.846285 2570 generic.go:356] "Generic (PLEG): container finished" podID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerID="429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3" exitCode=0 Jul 31 23:50:57.846438 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.846358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerDied","Data":"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3"} Jul 31 23:50:57.846719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.846703 2570 scope.go:117] "RemoveContainer" containerID="429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3" Jul 31 23:50:57.847803 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.847772 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:57.847938 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.847874 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:57.848004 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:57.847939 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:50:58.858113 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:58.858077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerStarted","Data":"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3"} Jul 31 23:50:58.945250 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:58.945208 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:50:58.945521 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:58.945494 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:50:58.945609 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:58.945553 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:50:59.945476 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:59.945029 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:50:59.945476 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:50:59.945101 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:51:00.945454 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:00.945024 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:51:00.945454 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:00.945084 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:51:01.945330 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:01.945293 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:51:01.945617 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:01.945578 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:51:02.945567 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:02.945530 2570 patch_prober.go:28] interesting pod/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 31 23:51:02.946552 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:02.946139 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 31 23:51:03.949342 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:03.949312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:51:03.949869 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:03.949784 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:51:03.950475 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:03.950455 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:51:08.629105 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.629064 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:51:08.631476 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.631457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.634088 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.634068 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5a93d4-openshift-default-dockercfg-88fbt\"" Jul 31 23:51:08.647780 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.647759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:51:08.745100 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745108 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745267 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsfb\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745552 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745552 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.745552 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.745354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852276 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852375 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852441 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsfb\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852719 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.852913 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.853048 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.853048 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.852998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.853246 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.853180 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.853314 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.853248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.853727 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.853701 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.855039 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.855016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.855846 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.855811 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.861928 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.861903 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.862337 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.862316 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsfb\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb\") pod \"e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:08.942562 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:08.942536 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:09.080885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.080861 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:51:09.082573 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:51:09.082545 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a4d4f0_168a_4362_bdaf_2c4bd9342b90.slice/crio-c13da217ce186a5b16afff8c416c6ce22b125a64985f4023764825c6351fd80d WatchSource:0}: Error finding container c13da217ce186a5b16afff8c416c6ce22b125a64985f4023764825c6351fd80d: Status 404 returned error can't find the container with id c13da217ce186a5b16afff8c416c6ce22b125a64985f4023764825c6351fd80d Jul 31 23:51:09.084770 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.084735 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:09.084885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.084799 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:09.084885 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.084846 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:09.914709 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.914678 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerStarted","Data":"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf"} Jul 31 23:51:09.915087 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.914730 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerStarted","Data":"c13da217ce186a5b16afff8c416c6ce22b125a64985f4023764825c6351fd80d"} Jul 31 23:51:09.937023 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.936975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podStartSLOduration=1.93695987 podStartE2EDuration="1.93695987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:09.93566394 +0000 UTC m=+3257.857931733" watchObservedRunningTime="2026-07-31 23:51:09.93695987 +0000 UTC m=+3257.859227641" Jul 31 23:51:09.942747 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:09.942727 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:10.947896 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:10.947868 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:11.925456 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:11.925424 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:11.926479 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:11.926456 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:16.196964 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:16.196931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:51:26.016631 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.016552 2570 generic.go:356] "Generic (PLEG): container finished" podID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerID="21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf" exitCode=0 Jul 31 23:51:26.017150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.016632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerDied","Data":"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf"} Jul 31 23:51:26.017150 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.017092 2570 scope.go:117] "RemoveContainer" containerID="21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf" Jul 31 23:51:26.018221 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.018195 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:26.018292 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.018261 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:26.018329 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.018304 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:51:26.925366 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:26.925320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:27.024663 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:27.024627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerStarted","Data":"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb"} Jul 31 23:51:27.944083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:27.943999 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:27.944289 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:27.944192 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 31 23:51:27.944289 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:27.944249 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 31 23:51:28.943161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:28.943122 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 31 23:51:28.943736 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:28.943703 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 31 23:51:29.943369 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:29.943251 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 31 23:51:29.943369 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:29.943315 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 31 23:51:30.942946 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:30.942901 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 31 23:51:30.943157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:30.942976 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 31 23:51:31.949210 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:31.949180 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:31.949706 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:31.949576 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:31.950427 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:31.950408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:51:52.794531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:52.794505 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:51:52.798716 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:52.798695 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:51:52.806565 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:52.806545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:51:52.810330 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:51:52.810313 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nv77q_23ac3832-68eb-46ab-910a-d87d76403e76/ovn-acl-logging/0.log" Jul 31 23:52:00.820469 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:00.820431 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:52:27.587191 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:27.587150 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:52:39.287461 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.287427 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:52:39.290971 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.290953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.293420 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.293402 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-74ebf4-openshift-default-dockercfg-ls9ck\"" Jul 31 23:52:39.303324 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.303305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:52:39.456074 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456204 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456371 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456371 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456371 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.456371 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.456316 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxx6q\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557525 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557525 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557525 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxx6q\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.557804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.558164 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.557960 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.558164 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.558021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.558164 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.558078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.558333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.558258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.558597 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.558572 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.560001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.559980 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.560138 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.560120 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.574115 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.574080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxx6q\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.574709 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.574686 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token\") pod \"e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.602673 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.602650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:39.733694 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.733669 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:52:39.735238 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:52:39.735207 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7836a16a_55e5_4915_9f5c_389f84892a54.slice/crio-63e586a5c5e026b1e1628073c86a097a081452523c7f9d7ea49ef0dd8a376418 WatchSource:0}: Error finding container 63e586a5c5e026b1e1628073c86a097a081452523c7f9d7ea49ef0dd8a376418: Status 404 returned error can't find the container with id 63e586a5c5e026b1e1628073c86a097a081452523c7f9d7ea49ef0dd8a376418 Jul 31 23:52:39.737432 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.737395 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:39.737545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.737478 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:39.737545 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:39.737526 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:40.425696 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:40.425662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerStarted","Data":"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8"} Jul 31 23:52:40.426160 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:40.425710 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerStarted","Data":"63e586a5c5e026b1e1628073c86a097a081452523c7f9d7ea49ef0dd8a376418"} Jul 31 23:52:40.446240 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:40.446194 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podStartSLOduration=1.4461809030000001 podStartE2EDuration="1.446180903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:40.444104377 +0000 UTC m=+3348.366372170" watchObservedRunningTime="2026-07-31 23:52:40.446180903 +0000 UTC m=+3348.368448675" Jul 31 23:52:40.603008 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:40.602978 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:40.607699 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:40.607675 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:41.431596 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:41.431565 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:41.432447 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:41.432425 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:49.100077 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:49.100042 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:52:57.473988 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:57.473947 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 31 23:52:57.474369 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:57.474013 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 31 23:52:58.521862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.521816 2570 generic.go:356] "Generic (PLEG): container finished" podID="7836a16a-55e5-4915-9f5c-389f84892a54" containerID="c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8" exitCode=0 Jul 31 23:52:58.521862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.521849 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerDied","Data":"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8"} Jul 31 23:52:58.522234 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.522218 2570 scope.go:117] "RemoveContainer" containerID="c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8" Jul 31 23:52:58.523218 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.523191 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:58.523282 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.523259 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:58.523319 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:58.523293 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 31 23:52:59.530300 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:59.529238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerStarted","Data":"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973"} Jul 31 23:52:59.603817 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:59.603779 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:52:59.604048 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:59.604024 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 31 23:52:59.604137 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:52:59.604080 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 31 23:53:00.604028 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:00.603997 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 31 23:53:00.604406 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:00.604047 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 31 23:53:01.603622 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:01.603589 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 31 23:53:01.603795 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:01.603641 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 31 23:53:02.603325 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:02.603286 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 31 23:53:02.603718 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:02.603353 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 31 23:53:03.603641 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:03.603607 2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 31 23:53:03.604124 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:03.603692 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 31 23:53:04.607251 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:04.607224 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:53:04.607627 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:04.607602 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:53:04.608226 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:04.608210 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:53:08.085069 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.085034 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87"] Jul 31 23:53:08.087330 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.087314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:08.092332 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.092313 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:53:08.095412 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.095394 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:53:08.101202 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.101180 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87"] Jul 31 23:53:08.206725 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.206698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87\" (UID: \"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:08.307766 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.307583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87\" (UID: \"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:08.313564 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.308559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87\" (UID: \"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:08.397593 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.397500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:08.525613 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.525575 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87"] Jul 31 23:53:08.527160 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:53:08.527132 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6baf4a_ef3d_4c0a_a341_0d6d1f8a4e93.slice/crio-a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c WatchSource:0}: Error finding container a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c: Status 404 returned error can't find the container with id a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c Jul 31 23:53:08.585039 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:08.585014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" event={"ID":"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93","Type":"ContainerStarted","Data":"a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c"} Jul 31 23:53:09.592949 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:09.592858 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" containerID="b6059dc1bde44ff2e4cb3221c297195afa371605bf0c448cf69944b66f9cb79e" exitCode=0 Jul 31 23:53:09.593281 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:09.592948 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" event={"ID":"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93","Type":"ContainerDied","Data":"b6059dc1bde44ff2e4cb3221c297195afa371605bf0c448cf69944b66f9cb79e"} Jul 31 23:53:10.725253 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:10.725226 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:10.829560 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:10.829534 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca\") pod \"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93\" (UID: \"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93\") " Jul 31 23:53:10.829899 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:10.829876 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" (UID: "3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:53:10.930165 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:10.930107 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:11.604862 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:11.604806 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" Jul 31 23:53:11.605033 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:11.604862 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87" event={"ID":"3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93","Type":"ContainerDied","Data":"a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c"} Jul 31 23:53:11.605033 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:11.604910 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a837458b6a8e35f14058500140aa75f226f44af1eba2f935f077ff3b46d8228c" Jul 31 23:53:14.364704 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:14.364664 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87"] Jul 31 23:53:14.368143 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:14.368117 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-74ebf4-xss87"] Jul 31 23:53:14.539572 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:14.539536 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" path="/var/lib/kubelet/pods/3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93/volumes" Jul 31 23:53:17.296119 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:17.296084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:53:20.739679 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:20.739647 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:53:20.740118 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:20.739853 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" containerID="cri-o://38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973" gracePeriod=30 Jul 31 23:53:24.334588 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.334547 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk"] Jul 31 23:53:24.334983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.334903 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" containerName="revoke-keys" Jul 31 23:53:24.334983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.334915 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" containerName="revoke-keys" Jul 31 23:53:24.334983 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.334979 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6baf4a-ef3d-4c0a-a341-0d6d1f8a4e93" containerName="revoke-keys" Jul 31 23:53:24.337106 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.337088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:24.339505 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.339479 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:53:24.339617 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.339482 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:53:24.344944 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.344921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk"] Jul 31 23:53:24.432616 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.432595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk\" (UID: \"ff04e36c-ac03-4516-b6ac-8b186864fbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:24.533083 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.533053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk\" (UID: \"ff04e36c-ac03-4516-b6ac-8b186864fbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:24.533655 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.533636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk\" (UID: \"ff04e36c-ac03-4516-b6ac-8b186864fbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:24.648740 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.648679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:24.777851 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:24.777805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk"] Jul 31 23:53:24.778089 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:53:24.777862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff04e36c_ac03_4516_b6ac_8b186864fbf5.slice/crio-11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379 WatchSource:0}: Error finding container 11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379: Status 404 returned error can't find the container with id 11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379 Jul 31 23:53:25.683485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:25.683404 2570 generic.go:356] "Generic (PLEG): container finished" podID="ff04e36c-ac03-4516-b6ac-8b186864fbf5" containerID="a91b4e72ead3f732e2603241367d676864537ff5a5e66b68bc535962da111623" exitCode=0 Jul 31 23:53:25.683852 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:25.683488 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" event={"ID":"ff04e36c-ac03-4516-b6ac-8b186864fbf5","Type":"ContainerDied","Data":"a91b4e72ead3f732e2603241367d676864537ff5a5e66b68bc535962da111623"} Jul 31 23:53:25.683852 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:25.683538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" event={"ID":"ff04e36c-ac03-4516-b6ac-8b186864fbf5","Type":"ContainerStarted","Data":"11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379"} Jul 31 23:53:25.986646 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:25.986623 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:53:26.145863 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.145835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.145871 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.145895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.145919 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146001 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.145982 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146007 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxx6q\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146035 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146094 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146116 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket\") pod \"7836a16a-55e5-4915-9f5c-389f84892a54\" (UID: \"7836a16a-55e5-4915-9f5c-389f84892a54\") " Jul 31 23:53:26.146219 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146200 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data" (OuterVolumeSpecName: "istio-data") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:53:26.146485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146389 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:53:26.146485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146410 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-data\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.146485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146405 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:53:26.146485 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146468 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:53:26.146694 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.146638 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:53:26.148053 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.148023 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 23:53:26.148161 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.148097 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:53:26.148606 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.148578 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q" (OuterVolumeSpecName: "kube-api-access-rxx6q") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "kube-api-access-rxx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:53:26.148606 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.148584 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token" (OuterVolumeSpecName: "istio-token") pod "7836a16a-55e5-4915-9f5c-389f84892a54" (UID: "7836a16a-55e5-4915-9f5c-389f84892a54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:53:26.247140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247077 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-istio-envoy\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247103 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-istio-token\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247117 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxx6q\" (UniqueName: \"kubernetes.io/projected/7836a16a-55e5-4915-9f5c-389f84892a54-kube-api-access-rxx6q\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247129 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247142 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7836a16a-55e5-4915-9f5c-389f84892a54-istiod-ca-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247154 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-credential-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247165 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7836a16a-55e5-4915-9f5c-389f84892a54-workload-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.247333 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.247176 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7836a16a-55e5-4915-9f5c-389f84892a54-istio-podinfo\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:26.691175 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.691145 2570 generic.go:356] "Generic (PLEG): container finished" podID="7836a16a-55e5-4915-9f5c-389f84892a54" containerID="38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973" exitCode=0 Jul 31 23:53:26.691498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.691193 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" Jul 31 23:53:26.691498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.691233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerDied","Data":"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973"} Jul 31 23:53:26.691498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.691287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2" event={"ID":"7836a16a-55e5-4915-9f5c-389f84892a54","Type":"ContainerDied","Data":"63e586a5c5e026b1e1628073c86a097a081452523c7f9d7ea49ef0dd8a376418"} Jul 31 23:53:26.691498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.691318 2570 scope.go:117] "RemoveContainer" containerID="38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973" Jul 31 23:53:26.700947 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.700921 2570 scope.go:117] "RemoveContainer" containerID="c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8" Jul 31 23:53:26.711169 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.711079 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:53:26.711408 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.711317 2570 scope.go:117] "RemoveContainer" containerID="38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973" Jul 31 23:53:26.711889 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:53:26.711860 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973\": container with ID starting with 38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973 not found: ID does not exist" containerID="38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973" Jul 31 23:53:26.711988 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.711891 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973"} err="failed to get container status \"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973\": rpc error: code = NotFound desc = could not find container \"38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973\": container with ID starting with 38df419b9a718ca60a060c65e42f4bb3b53365143aee175af821776f291ac973 not found: ID does not exist" Jul 31 23:53:26.711988 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.711914 2570 scope.go:117] "RemoveContainer" containerID="c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8" Jul 31 23:53:26.712195 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:53:26.712175 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8\": container with ID starting with c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8 not found: ID does not exist" containerID="c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8" Jul 31 23:53:26.712288 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.712199 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8"} err="failed to get container status \"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8\": rpc error: code = NotFound desc = could not find container \"c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8\": container with ID starting with c84a0aa7aca2af783cea457f2e462f0cbd93d3fe2eae15c084a41ca10d415fc8 not found: ID does not exist" Jul 31 23:53:26.715373 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.715350 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-74ebf4-openshift-default-7c4f5954fb-sjmt2"] Jul 31 23:53:26.822735 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.822712 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:26.952777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.952747 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca\") pod \"ff04e36c-ac03-4516-b6ac-8b186864fbf5\" (UID: \"ff04e36c-ac03-4516-b6ac-8b186864fbf5\") " Jul 31 23:53:26.953140 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:26.953114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff04e36c-ac03-4516-b6ac-8b186864fbf5" (UID: "ff04e36c-ac03-4516-b6ac-8b186864fbf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:53:27.053925 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:27.053891 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04e36c-ac03-4516-b6ac-8b186864fbf5-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:53:27.698125 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:27.698088 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" Jul 31 23:53:27.698125 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:27.698098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk" event={"ID":"ff04e36c-ac03-4516-b6ac-8b186864fbf5","Type":"ContainerDied","Data":"11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379"} Jul 31 23:53:27.698657 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:27.698142 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f616c1f0d9628fb694e3d01cd2db36997bfee1e33ac89cee5c75e2b666a379" Jul 31 23:53:28.538201 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:28.538165 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" path="/var/lib/kubelet/pods/7836a16a-55e5-4915-9f5c-389f84892a54/volumes" Jul 31 23:53:37.803804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:37.803772 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:53:39.946685 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:39.946627 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk"] Jul 31 23:53:39.949288 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:39.949253 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c16217-p8jdk"] Jul 31 23:53:40.538863 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:40.538807 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff04e36c-ac03-4516-b6ac-8b186864fbf5" path="/var/lib/kubelet/pods/ff04e36c-ac03-4516-b6ac-8b186864fbf5/volumes" Jul 31 23:53:46.990436 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:53:46.990347 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:54:28.242799 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.242764 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:54:28.243515 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.243019 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" containerID="cri-o://c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3" gracePeriod=30 Jul 31 23:54:28.595273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595196 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq"] Jul 31 23:54:28.595498 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595487 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:28.595541 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595500 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:28.595541 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595524 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff04e36c-ac03-4516-b6ac-8b186864fbf5" containerName="revoke-keys" Jul 31 23:54:28.595541 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595530 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff04e36c-ac03-4516-b6ac-8b186864fbf5" containerName="revoke-keys" Jul 31 23:54:28.595638 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595581 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:28.595638 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595590 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:28.595638 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.595599 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff04e36c-ac03-4516-b6ac-8b186864fbf5" containerName="revoke-keys" Jul 31 23:54:28.598273 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.598258 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:28.600764 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.600743 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:54:28.600903 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.600772 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:54:28.605236 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.605210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq"] Jul 31 23:54:28.736420 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.736393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq\" (UID: \"a493181b-ae4a-4b3d-b0c8-11a006ffac8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:28.836873 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.836846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq\" (UID: \"a493181b-ae4a-4b3d-b0c8-11a006ffac8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:28.837380 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.837363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq\" (UID: \"a493181b-ae4a-4b3d-b0c8-11a006ffac8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:28.910028 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:28.909970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:29.249539 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:29.249507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq"] Jul 31 23:54:29.250125 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:54:29.250098 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda493181b_ae4a_4b3d_b0c8_11a006ffac8e.slice/crio-6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0 WatchSource:0}: Error finding container 6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0: Status 404 returned error can't find the container with id 6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0 Jul 31 23:54:30.038802 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:30.038767 2570 generic.go:356] "Generic (PLEG): container finished" podID="a493181b-ae4a-4b3d-b0c8-11a006ffac8e" containerID="1c41a3436881ba3ca3ba97cbc434ac68d19f3cd0c04c965bd7049aef3a5964c4" exitCode=0 Jul 31 23:54:30.038928 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:30.038846 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" event={"ID":"a493181b-ae4a-4b3d-b0c8-11a006ffac8e","Type":"ContainerDied","Data":"1c41a3436881ba3ca3ba97cbc434ac68d19f3cd0c04c965bd7049aef3a5964c4"} Jul 31 23:54:30.038928 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:30.038893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" event={"ID":"a493181b-ae4a-4b3d-b0c8-11a006ffac8e","Type":"ContainerStarted","Data":"6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0"} Jul 31 23:54:31.174159 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:31.174132 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:31.253252 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:31.253226 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca\") pod \"a493181b-ae4a-4b3d-b0c8-11a006ffac8e\" (UID: \"a493181b-ae4a-4b3d-b0c8-11a006ffac8e\") " Jul 31 23:54:31.253557 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:31.253535 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a493181b-ae4a-4b3d-b0c8-11a006ffac8e" (UID: "a493181b-ae4a-4b3d-b0c8-11a006ffac8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:54:31.354092 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:31.354024 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a493181b-ae4a-4b3d-b0c8-11a006ffac8e-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:32.049714 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:32.049688 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" Jul 31 23:54:32.049714 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:32.049700 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq" event={"ID":"a493181b-ae4a-4b3d-b0c8-11a006ffac8e","Type":"ContainerDied","Data":"6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0"} Jul 31 23:54:32.049937 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:32.049750 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dbf66e51e9e1603261d07b83d25134e962bd0918e42af0fe945561cf7f95fa0" Jul 31 23:54:33.499354 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.499332 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:54:33.569868 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.569775 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.569999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.569889 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.569999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.569915 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.569999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.569957 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.569999 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.569989 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7xs5\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.570207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570002 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:33.570207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570016 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.570207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570049 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.570207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570076 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.570207 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570101 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo\") pod \"b2528601-3938-4ff0-9534-4068f56fc6ee\" (UID: \"b2528601-3938-4ff0-9534-4068f56fc6ee\") " Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570205 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570289 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570381 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570406 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-data\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570420 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-workload-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.570463 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570379 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:54:33.570745 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.570647 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:33.571996 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.571970 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:33.572129 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.572114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 23:54:33.572184 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.572164 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5" (OuterVolumeSpecName: "kube-api-access-f7xs5") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "kube-api-access-f7xs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:54:33.572260 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.572233 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2528601-3938-4ff0-9534-4068f56fc6ee" (UID: "b2528601-3938-4ff0-9534-4068f56fc6ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:54:33.671742 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671717 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-envoy\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.671902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671744 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xs5\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-kube-api-access-f7xs5\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.671902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671760 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2528601-3938-4ff0-9534-4068f56fc6ee-istiod-ca-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.671902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671770 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2528601-3938-4ff0-9534-4068f56fc6ee-credential-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.671902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671778 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-token\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:33.671902 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:33.671786 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2528601-3938-4ff0-9534-4068f56fc6ee-istio-podinfo\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:34.068224 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.068195 2570 generic.go:356] "Generic (PLEG): container finished" podID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerID="c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3" exitCode=0 Jul 31 23:54:34.068379 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.068254 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" Jul 31 23:54:34.068379 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.068285 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerDied","Data":"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3"} Jul 31 23:54:34.068379 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.068372 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff" event={"ID":"b2528601-3938-4ff0-9534-4068f56fc6ee","Type":"ContainerDied","Data":"b828a283d940832dc534748c4ede934b336b7d4c9b3c991078ee28790d6baa85"} Jul 31 23:54:34.068526 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.068404 2570 scope.go:117] "RemoveContainer" containerID="c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3" Jul 31 23:54:34.081025 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.080993 2570 scope.go:117] "RemoveContainer" containerID="429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3" Jul 31 23:54:34.090396 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.090368 2570 scope.go:117] "RemoveContainer" containerID="c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3" Jul 31 23:54:34.090692 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:54:34.090669 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3\": container with ID starting with c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3 not found: ID does not exist" containerID="c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3" Jul 31 23:54:34.090765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.090703 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3"} err="failed to get container status \"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3\": rpc error: code = NotFound desc = could not find container \"c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3\": container with ID starting with c4aeb91695e0f703645de23ecb731cf5b8e52e68a5f91f3a2f75c2b2ad21acc3 not found: ID does not exist" Jul 31 23:54:34.090765 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.090723 2570 scope.go:117] "RemoveContainer" containerID="429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3" Jul 31 23:54:34.090994 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:54:34.090978 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3\": container with ID starting with 429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3 not found: ID does not exist" containerID="429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3" Jul 31 23:54:34.091057 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.091011 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3"} err="failed to get container status \"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3\": rpc error: code = NotFound desc = could not find container \"429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3\": container with ID starting with 429ac9dc8521f745fcb301f9d29aad93dfc3fb745c8b519ad56045a9cb42bad3 not found: ID does not exist" Jul 31 23:54:34.092542 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.092520 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:54:34.096205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.096185 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c16217-openshift-default-8679d94b-n6qff"] Jul 31 23:54:34.538133 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:34.538100 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" path="/var/lib/kubelet/pods/b2528601-3938-4ff0-9534-4068f56fc6ee/volumes" Jul 31 23:54:39.311993 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:39.311960 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq"] Jul 31 23:54:39.315656 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:39.315629 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a93d4-ssvgq"] Jul 31 23:54:40.543974 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:40.543936 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a493181b-ae4a-4b3d-b0c8-11a006ffac8e" path="/var/lib/kubelet/pods/a493181b-ae4a-4b3d-b0c8-11a006ffac8e/volumes" Jul 31 23:54:41.485495 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:41.485457 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:54:45.569261 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.569226 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:54:45.569709 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.569475 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" containerID="cri-o://befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb" gracePeriod=30 Jul 31 23:54:45.934283 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934196 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45"] Jul 31 23:54:45.934598 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934579 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.934684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934602 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.934684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934626 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.934684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934635 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.934684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934659 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a493181b-ae4a-4b3d-b0c8-11a006ffac8e" containerName="revoke-keys" Jul 31 23:54:45.934684 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934670 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a493181b-ae4a-4b3d-b0c8-11a006ffac8e" containerName="revoke-keys" Jul 31 23:54:45.934975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934688 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:45.934975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934696 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7836a16a-55e5-4915-9f5c-389f84892a54" containerName="istio-proxy" Jul 31 23:54:45.934975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934783 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a493181b-ae4a-4b3d-b0c8-11a006ffac8e" containerName="revoke-keys" Jul 31 23:54:45.934975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934799 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.934975 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.934813 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2528601-3938-4ff0-9534-4068f56fc6ee" containerName="istio-proxy" Jul 31 23:54:45.938026 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.938005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:45.940698 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.940675 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:54:45.940997 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.940779 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:54:45.943968 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:45.943944 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45"] Jul 31 23:54:46.063898 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.063864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45\" (UID: \"7686519b-fbf1-47d4-a777-7db1236f6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:46.164434 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.164389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45\" (UID: \"7686519b-fbf1-47d4-a777-7db1236f6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:46.165041 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.165022 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45\" (UID: \"7686519b-fbf1-47d4-a777-7db1236f6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:46.250085 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.250058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:46.378633 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.378609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45"] Jul 31 23:54:46.380333 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:54:46.380303 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7686519b_fbf1_47d4_a777_7db1236f6888.slice/crio-4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3 WatchSource:0}: Error finding container 4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3: Status 404 returned error can't find the container with id 4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3 Jul 31 23:54:46.951609 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.951581 2570 patch_prober.go:28] interesting pod/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 31 23:54:46.951936 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:46.951630 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 23:54:47.140570 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:47.140536 2570 generic.go:356] "Generic (PLEG): container finished" podID="7686519b-fbf1-47d4-a777-7db1236f6888" containerID="820bfc854c970558fe14a1a0c093853f7e677ec190b27c83b56e5c8a783c0dfe" exitCode=0 Jul 31 23:54:47.140743 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:47.140625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" event={"ID":"7686519b-fbf1-47d4-a777-7db1236f6888","Type":"ContainerDied","Data":"820bfc854c970558fe14a1a0c093853f7e677ec190b27c83b56e5c8a783c0dfe"} Jul 31 23:54:47.140743 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:47.140694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" event={"ID":"7686519b-fbf1-47d4-a777-7db1236f6888","Type":"ContainerStarted","Data":"4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3"} Jul 31 23:54:48.272650 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:48.272627 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:48.384168 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:48.384138 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca\") pod \"7686519b-fbf1-47d4-a777-7db1236f6888\" (UID: \"7686519b-fbf1-47d4-a777-7db1236f6888\") " Jul 31 23:54:48.384512 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:48.384491 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7686519b-fbf1-47d4-a777-7db1236f6888" (UID: "7686519b-fbf1-47d4-a777-7db1236f6888"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:54:48.485066 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:48.485041 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7686519b-fbf1-47d4-a777-7db1236f6888-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:49.152365 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:49.152337 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" Jul 31 23:54:49.152526 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:49.152370 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45" event={"ID":"7686519b-fbf1-47d4-a777-7db1236f6888","Type":"ContainerDied","Data":"4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3"} Jul 31 23:54:49.152526 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:49.152413 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4885a67ea30ed201b0df7896f31a8dc2648a7a25c892250f3616f8ba8e431ed3" Jul 31 23:54:50.809467 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.809444 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:54:50.905379 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905312 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905369 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zsfb\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905409 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905432 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905448 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905506 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905480 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905623 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905777 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905663 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs\") pod \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\" (UID: \"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90\") " Jul 31 23:54:50.905965 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905786 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:50.905965 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.905795 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:50.906139 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.906019 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:50.906139 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.906045 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:50.906326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.906026 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:50.906326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.906162 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:54:50.906326 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.906174 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-data\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:50.907683 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.907655 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb" (OuterVolumeSpecName: "kube-api-access-8zsfb") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "kube-api-access-8zsfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:54:50.907683 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.907670 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 23:54:50.907809 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.907681 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 23:54:50.907809 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:50.907693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" (UID: "f4a4d4f0-168a-4362-bdaf-2c4bd9342b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 31 23:54:51.007090 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007061 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zsfb\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-kube-api-access-8zsfb\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007090 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007086 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-credential-socket\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007099 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-podinfo\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007112 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-envoy\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007123 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istiod-ca-cert\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007135 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-workload-certs\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.007231 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.007146 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90-istio-token\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:54:51.165875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.165794 2570 generic.go:356] "Generic (PLEG): container finished" podID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerID="befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb" exitCode=0 Jul 31 23:54:51.165875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.165864 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" Jul 31 23:54:51.166002 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.165880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerDied","Data":"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb"} Jul 31 23:54:51.166002 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.165936 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq" event={"ID":"f4a4d4f0-168a-4362-bdaf-2c4bd9342b90","Type":"ContainerDied","Data":"c13da217ce186a5b16afff8c416c6ce22b125a64985f4023764825c6351fd80d"} Jul 31 23:54:51.166002 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.165960 2570 scope.go:117] "RemoveContainer" containerID="befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb" Jul 31 23:54:51.178671 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.178654 2570 scope.go:117] "RemoveContainer" containerID="21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf" Jul 31 23:54:51.186860 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.186836 2570 scope.go:117] "RemoveContainer" containerID="befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb" Jul 31 23:54:51.187107 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:54:51.187089 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb\": container with ID starting with befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb not found: ID does not exist" containerID="befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb" Jul 31 23:54:51.187187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.187119 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb"} err="failed to get container status \"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb\": rpc error: code = NotFound desc = could not find container \"befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb\": container with ID starting with befb68b4b2278ab3bcb3425e2eecd9d49cd74195db30161f5da0ca1011c920bb not found: ID does not exist" Jul 31 23:54:51.187187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.187144 2570 scope.go:117] "RemoveContainer" containerID="21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf" Jul 31 23:54:51.187372 ip-10-0-137-58 kubenswrapper[2570]: E0731 23:54:51.187356 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf\": container with ID starting with 21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf not found: ID does not exist" containerID="21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf" Jul 31 23:54:51.187431 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.187379 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf"} err="failed to get container status \"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf\": rpc error: code = NotFound desc = could not find container \"21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf\": container with ID starting with 21c32a2e1ed0d4aa5efccbffaa6948f7d0e3bf5580a784cf66f16579a28ec4bf not found: ID does not exist" Jul 31 23:54:51.190589 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.190567 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:54:51.193907 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:51.193888 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5a93d4-openshift-default-5fd65ffc6-gtpzq"] Jul 31 23:54:52.538523 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:52.538492 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" path="/var/lib/kubelet/pods/f4a4d4f0-168a-4362-bdaf-2c4bd9342b90/volumes" Jul 31 23:54:56.488430 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:56.488395 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc"] Jul 31 23:54:56.490741 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:56.490721 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99d0e4-29759025-s5jlc"] Jul 31 23:54:56.538483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:56.538449 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba270735-14c4-4075-b65e-9a193b8c5784" path="/var/lib/kubelet/pods/ba270735-14c4-4075-b65e-9a193b8c5784/volumes" Jul 31 23:54:56.650899 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:56.650871 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45"] Jul 31 23:54:56.654076 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:56.654053 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99d0e4-rgq45"] Jul 31 23:54:58.538446 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:58.538408 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7686519b-fbf1-47d4-a777-7db1236f6888" path="/var/lib/kubelet/pods/7686519b-fbf1-47d4-a777-7db1236f6888/volumes" Jul 31 23:54:58.887607 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:54:58.887527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:55:03.453762 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.453728 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb"] Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454051 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454064 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454089 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7686519b-fbf1-47d4-a777-7db1236f6888" containerName="revoke-keys" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454096 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7686519b-fbf1-47d4-a777-7db1236f6888" containerName="revoke-keys" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454144 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7686519b-fbf1-47d4-a777-7db1236f6888" containerName="revoke-keys" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454160 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:03.454216 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.454169 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:03.459832 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.459798 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:03.462982 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.462957 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qxwgn\"" Jul 31 23:55:03.463384 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.463361 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 23:55:03.464940 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.464914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb"] Jul 31 23:55:03.596298 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.596268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb\" (UID: \"b17822c6-54d1-4ab5-9142-ba323347eeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:03.697205 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.697177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb\" (UID: \"b17822c6-54d1-4ab5-9142-ba323347eeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:03.697742 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.697725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb\" (UID: \"b17822c6-54d1-4ab5-9142-ba323347eeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:03.772768 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:03.772714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:04.111191 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:04.111165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb"] Jul 31 23:55:04.112495 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:55:04.112470 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17822c6_54d1_4ab5_9142_ba323347eeb8.slice/crio-065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524 WatchSource:0}: Error finding container 065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524: Status 404 returned error can't find the container with id 065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524 Jul 31 23:55:04.236887 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:04.236858 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" event={"ID":"b17822c6-54d1-4ab5-9142-ba323347eeb8","Type":"ContainerStarted","Data":"065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524"} Jul 31 23:55:05.244157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:05.244121 2570 generic.go:356] "Generic (PLEG): container finished" podID="b17822c6-54d1-4ab5-9142-ba323347eeb8" containerID="2faec9464747b7f851ae8baf82f70daf1c83d4c14e37d281fe901170c67e57a1" exitCode=0 Jul 31 23:55:05.244568 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:05.244165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" event={"ID":"b17822c6-54d1-4ab5-9142-ba323347eeb8","Type":"ContainerDied","Data":"2faec9464747b7f851ae8baf82f70daf1c83d4c14e37d281fe901170c67e57a1"} Jul 31 23:55:06.383480 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:06.383454 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:06.516040 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:06.515974 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca\") pod \"b17822c6-54d1-4ab5-9142-ba323347eeb8\" (UID: \"b17822c6-54d1-4ab5-9142-ba323347eeb8\") " Jul 31 23:55:06.516314 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:06.516290 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b17822c6-54d1-4ab5-9142-ba323347eeb8" (UID: "b17822c6-54d1-4ab5-9142-ba323347eeb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 23:55:06.616570 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:06.616543 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b17822c6-54d1-4ab5-9142-ba323347eeb8-maas-api-service-ca\") on node \"ip-10-0-137-58.ec2.internal\" DevicePath \"\"" Jul 31 23:55:07.257054 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:07.257025 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" Jul 31 23:55:07.257227 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:07.257060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb" event={"ID":"b17822c6-54d1-4ab5-9142-ba323347eeb8","Type":"ContainerDied","Data":"065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524"} Jul 31 23:55:07.257227 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:07.257116 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="065af2dea8c26993977e4b39de21871d773eeca69cb62ebf753564c5a820a524" Jul 31 23:55:13.852041 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:13.852008 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8"] Jul 31 23:55:13.853834 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:13.853801 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3d60f7-29759025-f9dq8"] Jul 31 23:55:14.010307 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:14.010276 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb"] Jul 31 23:55:14.012970 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:14.012940 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3d60f7-bwjdb"] Jul 31 23:55:14.538670 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:14.538634 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2acd22-ee12-49d9-ab48-4c6dca92a863" path="/var/lib/kubelet/pods/2b2acd22-ee12-49d9-ab48-4c6dca92a863/volumes" Jul 31 23:55:14.539040 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:14.539020 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17822c6-54d1-4ab5-9142-ba323347eeb8" path="/var/lib/kubelet/pods/b17822c6-54d1-4ab5-9142-ba323347eeb8/volumes" Jul 31 23:55:15.992322 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:15.992246 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7rwtl"] Jul 31 23:55:21.090984 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:21.090958 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dd6b4dfd5-sc5fm_3fd9344b-7bb6-4f0a-8a21-94048704e45d/authorino/0.log" Jul 31 23:55:26.611430 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:26.611399 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-56v8k_8e934743-011d-4538-b7b1-2bb8e70c26e0/manager/0.log" Jul 31 23:55:26.959275 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:26.959246 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8567dd98f6-v6rxq_dc007521-9c05-4f43-99b6-90b02652eb03/copy-manifests/0.log" Jul 31 23:55:26.971388 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:26.971363 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8567dd98f6-v6rxq_dc007521-9c05-4f43-99b6-90b02652eb03/manager/0.log" Jul 31 23:55:27.525327 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:27.525300 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759010-j4xw5_900dadec-966d-499b-b786-e7ddc9e449bb/cleanup/0.log" Jul 31 23:55:27.636510 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:27.636476 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759025-v8zqk_5e3b97aa-9d5c-417f-99f3-4674adbc5f31/cleanup/0.log" Jul 31 23:55:28.518203 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:28.518173 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dd6b4dfd5-sc5fm_3fd9344b-7bb6-4f0a-8a21-94048704e45d/authorino/0.log" Jul 31 23:55:28.843187 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:28.843106 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-tl2nl_fff8e9da-2fcf-4414-be73-6c42763e12cc/kuadrant-console-plugin/0.log" Jul 31 23:55:29.076584 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:29.076554 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7rwtl_230d3ca1-40fe-405e-ab1b-cb745647b7f1/limitador/0.log" Jul 31 23:55:29.833283 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:29.833257 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85c4c7ff5b-t8lgl_2db0525c-dadd-4b88-95e5-7392a88126ad/kube-auth-proxy/0.log" Jul 31 23:55:36.769972 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.769935 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6dw7l/must-gather-4p25p"] Jul 31 23:55:36.770531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.770393 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17822c6-54d1-4ab5-9142-ba323347eeb8" containerName="revoke-keys" Jul 31 23:55:36.770531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.770413 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17822c6-54d1-4ab5-9142-ba323347eeb8" containerName="revoke-keys" Jul 31 23:55:36.770531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.770426 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:36.770531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.770437 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a4d4f0-168a-4362-bdaf-2c4bd9342b90" containerName="istio-proxy" Jul 31 23:55:36.770531 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.770519 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17822c6-54d1-4ab5-9142-ba323347eeb8" containerName="revoke-keys" Jul 31 23:55:36.774634 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.774613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.777247 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.777228 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6dw7l\"/\"kube-root-ca.crt\"" Jul 31 23:55:36.777358 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.777230 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6dw7l\"/\"openshift-service-ca.crt\"" Jul 31 23:55:36.778297 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.778281 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6dw7l\"/\"default-dockercfg-vwvnq\"" Jul 31 23:55:36.782297 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.782275 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dw7l/must-gather-4p25p"] Jul 31 23:55:36.840677 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.840651 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38288449-f565-4821-b2e9-80455afc6797-must-gather-output\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.840784 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.840693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjj8b\" (UniqueName: \"kubernetes.io/projected/38288449-f565-4821-b2e9-80455afc6797-kube-api-access-sjj8b\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.941647 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.941616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38288449-f565-4821-b2e9-80455afc6797-must-gather-output\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.941742 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.941662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjj8b\" (UniqueName: \"kubernetes.io/projected/38288449-f565-4821-b2e9-80455afc6797-kube-api-access-sjj8b\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.941968 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.941943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38288449-f565-4821-b2e9-80455afc6797-must-gather-output\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:36.949832 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:36.949802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjj8b\" (UniqueName: \"kubernetes.io/projected/38288449-f565-4821-b2e9-80455afc6797-kube-api-access-sjj8b\") pod \"must-gather-4p25p\" (UID: \"38288449-f565-4821-b2e9-80455afc6797\") " pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:37.084793 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:37.084740 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dw7l/must-gather-4p25p" Jul 31 23:55:37.213157 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:37.213132 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dw7l/must-gather-4p25p"] Jul 31 23:55:37.215113 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:55:37.215087 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38288449_f565_4821_b2e9_80455afc6797.slice/crio-0358112e162c62cc78c8389c946076d0531ae82d75036eca668789cb256c6633 WatchSource:0}: Error finding container 0358112e162c62cc78c8389c946076d0531ae82d75036eca668789cb256c6633: Status 404 returned error can't find the container with id 0358112e162c62cc78c8389c946076d0531ae82d75036eca668789cb256c6633 Jul 31 23:55:37.216896 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:37.216878 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 23:55:37.419721 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:37.419655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dw7l/must-gather-4p25p" event={"ID":"38288449-f565-4821-b2e9-80455afc6797","Type":"ContainerStarted","Data":"0358112e162c62cc78c8389c946076d0531ae82d75036eca668789cb256c6633"} Jul 31 23:55:38.428132 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:38.428040 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dw7l/must-gather-4p25p" event={"ID":"38288449-f565-4821-b2e9-80455afc6797","Type":"ContainerStarted","Data":"26621543fcc1b780721b927b8bf7cb482cf49dd672d0f23e5afa253af66f7b1c"} Jul 31 23:55:38.428132 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:38.428086 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dw7l/must-gather-4p25p" event={"ID":"38288449-f565-4821-b2e9-80455afc6797","Type":"ContainerStarted","Data":"472f0028ff628a434deea110bd911bebc5f5668ccd1c897e499bb07363e8be9c"} Jul 31 23:55:38.445724 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:38.445649 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6dw7l/must-gather-4p25p" podStartSLOduration=1.701158497 podStartE2EDuration="2.445620552s" totalImagesPullingTime="744.462055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:38.444010288 +0000 UTC m=+3526.366278082" watchObservedRunningTime="2026-07-31 23:55:38.445620552 +0000 UTC m=+3526.367888347" Jul 31 23:55:39.457249 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:39.457223 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rrsk_091a5632-061d-4b31-8e30-f03fffa6a68d/global-pull-secret-syncer/0.log" Jul 31 23:55:39.670194 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:39.670087 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vmspv_43cb94e5-0771-4262-bc66-258d46c52571/konnectivity-agent/0.log" Jul 31 23:55:39.741330 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:39.741259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-58.ec2.internal_a6ebc48e958d442fefbec257d10b60ea/haproxy/0.log" Jul 31 23:55:44.609844 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:44.606357 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dd6b4dfd5-sc5fm_3fd9344b-7bb6-4f0a-8a21-94048704e45d/authorino/0.log" Jul 31 23:55:44.711374 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:44.711268 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-tl2nl_fff8e9da-2fcf-4414-be73-6c42763e12cc/kuadrant-console-plugin/0.log" Jul 31 23:55:44.853881 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:44.853851 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7rwtl_230d3ca1-40fe-405e-ab1b-cb745647b7f1/limitador/0.log" Jul 31 23:55:46.945134 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:46.945099 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc5xk_0f1f8c22-a0c8-4777-b8f9-f5355a2bf892/node-exporter/0.log" Jul 31 23:55:46.969743 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:46.969712 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc5xk_0f1f8c22-a0c8-4777-b8f9-f5355a2bf892/kube-rbac-proxy/0.log" Jul 31 23:55:46.997190 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:46.997163 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc5xk_0f1f8c22-a0c8-4777-b8f9-f5355a2bf892/init-textfile/0.log" Jul 31 23:55:49.311230 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.311188 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj"] Jul 31 23:55:49.315806 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.315777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.334010 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.333973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj"] Jul 31 23:55:49.442044 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.442014 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/2.log" Jul 31 23:55:49.449742 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.449706 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hwdkz_78954292-2379-439c-babe-b86b2156e4a1/console-operator/3.log" Jul 31 23:55:49.473110 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.473018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-sys\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.473370 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.473350 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-lib-modules\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.473582 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.473534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-proc\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.473782 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.473745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-podres\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.473986 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.473950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cmrq\" (UniqueName: \"kubernetes.io/projected/5bd11612-7b58-400f-8510-d70609b36389-kube-api-access-5cmrq\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575396 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-sys\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575396 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-lib-modules\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575396 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-proc\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575693 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-podres\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575693 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cmrq\" (UniqueName: \"kubernetes.io/projected/5bd11612-7b58-400f-8510-d70609b36389-kube-api-access-5cmrq\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.575875 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575852 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-sys\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.576004 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.575987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-lib-modules\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.576070 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.576041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-proc\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.576128 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.576110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bd11612-7b58-400f-8510-d70609b36389-podres\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.584920 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.584892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cmrq\" (UniqueName: \"kubernetes.io/projected/5bd11612-7b58-400f-8510-d70609b36389-kube-api-access-5cmrq\") pod \"perf-node-gather-daemonset-p4nkj\" (UID: \"5bd11612-7b58-400f-8510-d70609b36389\") " pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.630432 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.630403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:49.802801 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:49.802717 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj"] Jul 31 23:55:49.807946 ip-10-0-137-58 kubenswrapper[2570]: W0731 23:55:49.807909 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5bd11612_7b58_400f_8510_d70609b36389.slice/crio-32dd718647f13ad876f21498a8dee82f8b560a760d8a40c91595eff1ea224190 WatchSource:0}: Error finding container 32dd718647f13ad876f21498a8dee82f8b560a760d8a40c91595eff1ea224190: Status 404 returned error can't find the container with id 32dd718647f13ad876f21498a8dee82f8b560a760d8a40c91595eff1ea224190 Jul 31 23:55:50.032763 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.032728 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84494b959b-mllr9_e29304de-3d07-4a1c-8ea9-c3d73f1bf551/console/0.log" Jul 31 23:55:50.072227 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.072186 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-rgtg5_a1ed22fe-e2ba-4afe-a48b-4a4dafadfdd9/download-server/0.log" Jul 31 23:55:50.546649 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.546613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" event={"ID":"5bd11612-7b58-400f-8510-d70609b36389","Type":"ContainerStarted","Data":"0e1cbebe378652ec84403d848ac3df4727b2bdf1460691cc80dfcb86d5a6cdb4"} Jul 31 23:55:50.547049 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.546683 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:50.547049 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.546709 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" event={"ID":"5bd11612-7b58-400f-8510-d70609b36389","Type":"ContainerStarted","Data":"32dd718647f13ad876f21498a8dee82f8b560a760d8a40c91595eff1ea224190"} Jul 31 23:55:50.562757 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.562699 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" podStartSLOduration=1.562679272 podStartE2EDuration="1.562679272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:50.562579314 +0000 UTC m=+3538.484847084" watchObservedRunningTime="2026-07-31 23:55:50.562679272 +0000 UTC m=+3538.484947022" Jul 31 23:55:50.743745 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:50.743708 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-wtf42_4c8f896c-6a8e-4f9c-8d4c-af1d5e4aefee/volume-data-source-validator/0.log" Jul 31 23:55:51.863286 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:51.863244 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pg6g5_1e665f6f-530c-4b2b-967b-8d4be419d54a/dns/0.log" Jul 31 23:55:51.882582 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:51.882538 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pg6g5_1e665f6f-530c-4b2b-967b-8d4be419d54a/kube-rbac-proxy/0.log" Jul 31 23:55:51.990251 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:51.990222 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzgjv_9fcb9bc0-23e8-4f43-9468-54fb1c39bd23/dns-node-resolver/0.log" Jul 31 23:55:52.516352 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:52.516325 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rfbw_7dd0530a-f47c-4900-95e6-e9fec3c7924d/node-ca/0.log" Jul 31 23:55:53.127224 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:53.127192 2570 scope.go:117] "RemoveContainer" containerID="82c734bc8e184d5057af030370b3735ae58d0bfb6cd692ce877412410a688d9f" Jul 31 23:55:53.139057 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:53.139032 2570 scope.go:117] "RemoveContainer" containerID="19d3c939692fc481124274860b37f40e35065cba2f9d047721b9bf9c149fee98" Jul 31 23:55:53.684768 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:53.684736 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85c4c7ff5b-t8lgl_2db0525c-dadd-4b88-95e5-7392a88126ad/kube-auth-proxy/0.log" Jul 31 23:55:54.570275 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:54.570239 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87fdb_50f9762d-6540-44e2-bddc-db63f798f2b5/serve-healthcheck-canary/0.log" Jul 31 23:55:55.125974 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:55.125947 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mw7k9_78518aa9-6d05-47ec-b30a-7c6fdb8f4c31/kube-rbac-proxy/0.log" Jul 31 23:55:55.144804 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:55.144779 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mw7k9_78518aa9-6d05-47ec-b30a-7c6fdb8f4c31/exporter/0.log" Jul 31 23:55:55.164479 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:55.164454 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mw7k9_78518aa9-6d05-47ec-b30a-7c6fdb8f4c31/extractor/0.log" Jul 31 23:55:56.566251 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:56.566224 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6dw7l/perf-node-gather-daemonset-p4nkj" Jul 31 23:55:57.777649 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:57.777457 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-56v8k_8e934743-011d-4538-b7b1-2bb8e70c26e0/manager/0.log" Jul 31 23:55:58.032967 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:58.032882 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8567dd98f6-v6rxq_dc007521-9c05-4f43-99b6-90b02652eb03/manager/0.log" Jul 31 23:55:58.075114 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:58.075085 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8567dd98f6-v6rxq_dc007521-9c05-4f43-99b6-90b02652eb03/copy-manifests/0.log" Jul 31 23:55:59.483941 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:55:59.483913 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6b79896bff-n26ql_ec1f1587-f293-49ef-9b3f-593321eebf51/manager/0.log" Jul 31 23:56:00.013389 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:00.013348 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85c4c7ff5b-t8lgl_2db0525c-dadd-4b88-95e5-7392a88126ad/kube-auth-proxy/0.log" Jul 31 23:56:01.695662 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:01.695633 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85c4c7ff5b-t8lgl_2db0525c-dadd-4b88-95e5-7392a88126ad/kube-auth-proxy/0.log" Jul 31 23:56:04.217670 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:04.217635 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-hfz22_fe6aaca1-cd56-4bb5-ac30-cabb3027fa58/migrator/0.log" Jul 31 23:56:04.239508 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:04.239477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-hfz22_fe6aaca1-cd56-4bb5-ac30-cabb3027fa58/graceful-termination/0.log" Jul 31 23:56:05.478372 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.478342 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/kube-multus-additional-cni-plugins/0.log" Jul 31 23:56:05.497358 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.497335 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/egress-router-binary-copy/0.log" Jul 31 23:56:05.516373 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.516349 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/cni-plugins/0.log" Jul 31 23:56:05.534988 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.534969 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/bond-cni-plugin/0.log" Jul 31 23:56:05.553469 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.553453 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/routeoverride-cni/0.log" Jul 31 23:56:05.571411 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.571395 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/whereabouts-cni-bincopy/0.log" Jul 31 23:56:05.634158 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.634138 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2rcc8_1871d052-b938-4d33-9aa5-cdad64a19843/whereabouts-cni/0.log" Jul 31 23:56:05.971028 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:05.970999 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dk5cv_0cc84881-0fc3-410e-bb1d-fe610a4c85a6/kube-multus/0.log" Jul 31 23:56:06.105163 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:06.105130 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9ftrs_842f999d-a623-4c09-8218-099a42fe84cc/network-metrics-daemon/0.log" Jul 31 23:56:06.142483 ip-10-0-137-58 kubenswrapper[2570]: I0731 23:56:06.142400 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9ftrs_842f999d-a623-4c09-8218-099a42fe84cc/kube-rbac-proxy/0.log"