Jul 10 21:18:57.702807 ip-10-0-130-213 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 21:18:57.703090 ip-10-0-130-213 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 21:18:57.703108 ip-10-0-130-213 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 21:18:57.703796 ip-10-0-130-213 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 21:19:08.373455 ip-10-0-130-213 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 10 21:19:08.373469 ip-10-0-130-213 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 22c4945e71604a62b7d361ffc11cb197 -- Jul 10 21:21:01.526422 ip-10-0-130-213 systemd[1]: Starting Kubernetes Kubelet... Jul 10 21:21:01.958693 ip-10-0-130-213 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 21:21:01.958693 ip-10-0-130-213 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 21:21:01.958693 ip-10-0-130-213 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 21:21:01.958693 ip-10-0-130-213 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 21:21:01.958693 ip-10-0-130-213 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 21:21:01.961666 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.961571 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 21:21:01.966121 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966099 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.966121 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966120 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.966121 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966124 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966127 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966132 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966135 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966140 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966142 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966145 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966148 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966151 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966153 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966156 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966159 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966164 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966168 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966171 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966174 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966176 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966179 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966181 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966184 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.966224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966187 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966189 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966192 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966194 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966197 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966199 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966202 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966204 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966207 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966211 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966213 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966215 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966218 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966222 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966227 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966230 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966234 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966236 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966239 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.966699 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966242 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966244 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966246 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966249 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966251 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966254 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966256 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966259 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966261 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966264 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966267 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966269 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966272 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966274 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966277 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966280 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966282 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966285 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966287 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966290 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.967283 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966292 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966295 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966297 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966299 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966302 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966305 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966307 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966311 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966314 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966317 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966319 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966322 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966326 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966328 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966331 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966333 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966336 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966338 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966341 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966343 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.967775 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966346 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966348 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966350 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966353 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966357 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966360 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966363 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966366 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966369 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966371 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966374 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966378 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966381 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966383 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966386 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966389 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966392 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.966394 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967162 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.968298 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967170 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967174 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967176 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967180 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967183 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967186 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967189 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967191 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967195 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967198 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967201 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967204 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967206 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967209 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967211 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967213 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967216 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967218 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967221 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967224 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.968791 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967227 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967229 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967232 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967234 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967237 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967240 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967243 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967245 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967248 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967251 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967253 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967256 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967258 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967261 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967265 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967269 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967272 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967276 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967279 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.969319 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967282 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967285 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967288 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967291 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967293 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967296 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967299 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967304 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967306 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967309 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967312 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967314 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967317 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967319 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967322 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967324 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967327 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967330 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967333 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.969793 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967335 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967338 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967340 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967343 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967345 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967348 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967350 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967352 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967355 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967358 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967360 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967363 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967368 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967372 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967375 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967377 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967379 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967382 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967384 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967387 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.970271 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967389 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967392 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967394 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967397 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967399 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967402 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967404 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967406 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967408 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967411 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967415 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967417 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967419 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967422 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967425 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967427 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967430 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967434 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967437 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967439 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.970762 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967886 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967892 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967895 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967898 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967902 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967905 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967908 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967910 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967913 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967915 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967918 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967923 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967928 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967931 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967933 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967936 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967938 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967941 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967945 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.971297 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967949 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967952 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967955 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967958 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967961 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967964 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967968 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967971 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967974 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967977 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967980 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967982 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967985 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967987 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967989 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967992 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967994 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.967997 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968001 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968004 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.971764 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968006 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968009 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968011 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968014 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968016 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968019 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968021 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968024 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968026 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968028 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968031 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968034 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968036 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968038 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968041 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968043 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968046 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968048 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968050 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968053 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.972275 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968055 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968075 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968078 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968081 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968083 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968086 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968088 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968091 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968093 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968096 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968098 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968101 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968104 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968107 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968109 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968112 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968114 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968117 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968119 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968122 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.972813 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968124 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968127 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968130 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968133 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968135 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968138 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968140 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968143 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968146 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968148 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968151 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968153 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968156 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968158 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968161 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968163 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968166 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968168 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968171 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968173 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.973318 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968257 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968269 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968285 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968291 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968299 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968303 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968307 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968316 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968319 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968322 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968325 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968328 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968331 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968335 2561 flags.go:64] FLAG: --cgroup-root="" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968338 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968341 2561 flags.go:64] FLAG: --client-ca-file="" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968344 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968347 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968352 2561 flags.go:64] FLAG: --cluster-domain="" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968355 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968358 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968361 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968364 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968368 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 21:21:01.973808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968373 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968376 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968379 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968383 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968386 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968389 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968392 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968395 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968399 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968402 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968405 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968407 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968410 2561 flags.go:64] FLAG: --enable-server="true" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968414 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968418 2561 flags.go:64] FLAG: --event-burst="100" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968421 2561 flags.go:64] FLAG: --event-qps="50" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968424 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968427 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968430 2561 flags.go:64] FLAG: --eviction-hard="" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968434 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968437 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968443 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968446 2561 flags.go:64] FLAG: --eviction-soft="" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968449 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968452 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 21:21:01.974428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968455 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968457 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968460 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968463 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968466 2561 flags.go:64] FLAG: --feature-gates="" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968470 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968473 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968478 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968481 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968484 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968487 2561 flags.go:64] FLAG: --help="false" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968490 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-213.ec2.internal" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968493 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968496 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968499 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968503 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968506 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968509 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968512 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968515 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968518 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968521 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968524 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968527 2561 flags.go:64] FLAG: --kube-reserved="" Jul 10 21:21:01.975185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968530 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968532 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968535 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968538 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968542 2561 flags.go:64] FLAG: --lock-file="" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968545 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968548 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968551 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968556 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968559 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968562 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968565 2561 flags.go:64] FLAG: --logging-format="text" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968567 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968571 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968573 2561 flags.go:64] FLAG: --manifest-url="" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968578 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968582 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968585 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968590 2561 flags.go:64] FLAG: --max-pods="110" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968593 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968596 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968599 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968601 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968604 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968607 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 21:21:01.976243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968610 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968618 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968621 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968624 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968627 2561 flags.go:64] FLAG: --pod-cidr="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968631 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968638 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968641 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968644 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968647 2561 flags.go:64] FLAG: --port="10250" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968650 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968653 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02693f349b69449d9" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968658 2561 flags.go:64] FLAG: --qos-reserved="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968661 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968664 2561 flags.go:64] FLAG: --register-node="true" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968666 2561 flags.go:64] FLAG: --register-with-taints="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968676 2561 flags.go:64] FLAG: --registry-burst="10" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968678 2561 flags.go:64] FLAG: --registry-qps="5" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968681 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968684 2561 flags.go:64] FLAG: --reserved-memory="" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968688 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968691 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968695 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968698 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968700 2561 flags.go:64] FLAG: --runonce="false" Jul 10 21:21:01.977334 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968703 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968706 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968709 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968712 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968715 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968717 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968721 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968724 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968726 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968729 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968732 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968735 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968738 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968742 2561 flags.go:64] FLAG: --system-cgroups="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968745 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968750 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968753 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968756 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968760 2561 flags.go:64] FLAG: --tls-min-version="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968766 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968769 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968772 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968775 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968778 2561 flags.go:64] FLAG: --v="2" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968782 2561 flags.go:64] FLAG: --version="false" Jul 10 21:21:01.978448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968786 2561 flags.go:64] FLAG: --vmodule="" Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968790 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.968793 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968913 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968918 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968921 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968924 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968927 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968929 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968933 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968935 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968939 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968941 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968944 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968947 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968950 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968952 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968955 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968957 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968960 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.979502 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968964 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968969 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968971 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968974 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968976 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968979 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968982 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968986 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968990 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968993 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968995 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.968998 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969000 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969003 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969005 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969007 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969011 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969014 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969016 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969019 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.980192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969021 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969024 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969026 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969029 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969032 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969034 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969037 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969039 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969042 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969044 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969047 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969049 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969052 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969054 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969057 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969074 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969077 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969080 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969084 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969087 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.980696 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969089 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969092 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969094 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969097 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969099 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969102 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969106 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969109 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969114 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969117 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969119 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969122 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969125 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969128 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969131 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969134 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969137 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969139 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969142 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.981372 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969145 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969147 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969150 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969153 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969156 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969158 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969161 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969163 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969166 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969169 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969172 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969175 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969178 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969181 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969183 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969185 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969188 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969190 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969193 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969195 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.981840 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969198 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969202 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.969204 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.969213 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.977259 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.977302 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.977332 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.977340 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977427 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977435 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977441 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977446 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977450 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977455 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977459 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.982337 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977464 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977468 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977473 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977477 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977481 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977486 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977490 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977495 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977499 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977503 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977507 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977511 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977515 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977519 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977524 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977530 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977536 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977541 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977546 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977550 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.982724 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977555 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977560 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977567 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977572 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977576 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977581 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977585 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977589 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977593 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977597 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977601 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977606 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977610 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977614 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977618 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977622 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977627 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977631 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977635 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.983225 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977639 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977644 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977648 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977652 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977656 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977660 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977664 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977669 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977673 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977678 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977682 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977686 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977690 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977694 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977721 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977728 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977734 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977739 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977743 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977749 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.983692 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977753 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977757 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977762 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977766 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977770 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977774 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977778 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977782 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977787 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977791 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977795 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977799 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977803 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977808 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977812 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977817 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977821 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977826 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977832 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.984243 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977838 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977843 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977848 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977853 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977858 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977862 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977867 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977871 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977878 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977884 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977890 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977895 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977899 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.977903 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.984723 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.977911 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978124 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978134 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978139 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978143 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978148 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978154 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978158 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978163 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978168 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978173 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978177 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978181 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978185 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978189 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978193 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978198 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978202 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978206 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978213 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 21:21:01.985144 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978220 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978225 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978230 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978235 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978239 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978243 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978248 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978252 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978256 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978261 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978265 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978270 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978274 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978278 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978282 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978286 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978290 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978295 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978299 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978303 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 21:21:01.985693 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978308 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978313 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978317 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978321 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978326 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978331 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978335 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978339 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978343 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978347 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978354 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978359 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978364 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978368 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978372 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978377 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978380 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978385 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978389 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 21:21:01.986274 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978394 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978398 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978402 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978407 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978412 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978415 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978419 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978423 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978427 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978431 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978435 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978439 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978443 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978447 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978451 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978454 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978458 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978462 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978466 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978469 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 21:21:01.986741 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978473 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978477 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978481 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978485 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978489 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978493 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978498 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978502 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978506 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978510 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978514 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978518 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978522 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978526 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978530 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978535 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978540 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978544 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978548 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978566 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 21:21:01.987378 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:01.978570 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.978579 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.978936 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.983768 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.985167 2561 server.go:1019] "Starting client certificate rotation" Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.985291 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.985935 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 21:21:01.988236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.987830 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 21:21:01.989049 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.989031 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 21:21:01.999133 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:01.999109 2561 log.go:25] "Validated CRI v1 runtime API" Jul 10 21:21:02.003574 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.003553 2561 log.go:25] "Validated CRI v1 image API" Jul 10 21:21:02.004413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.004398 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 21:21:02.007045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.007011 2561 fs.go:135] Filesystem UUIDs: map[62b1a076-6d6f-4c00-8ef2-6356938fd888:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b326da65-075f-436a-8d15-7493248a24be:/dev/nvme0n1p3] Jul 10 21:21:02.007149 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.007043 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 21:21:02.012379 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.012233 2561 manager.go:216] Machine: {Timestamp:2026-07-10 21:21:02.010765838 +0000 UTC m=+0.361289428 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100022 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b0fe6b7ecc9c4821ee82b04781ad SystemUUID:ec27b0fe-6b7e-cc9c-4821-ee82b04781ad BootID:22c4945e-7160-4a62-b7d3-61ffc11cb197 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:2b:79:7b:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:2b:79:7b:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:ed:9c:eb:f9:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 21:21:02.012379 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.012362 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 21:21:02.012571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.012480 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 21:21:02.013582 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.013547 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 21:21:02.013760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.013584 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-213.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 21:21:02.014352 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.014339 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 21:21:02.014413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.014357 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 21:21:02.014413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.014368 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 21:21:02.014413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.014383 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 21:21:02.015273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.015253 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 21:21:02.015324 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.015280 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 21:21:02.018398 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018381 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 10 21:21:02.018545 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018533 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 21:21:02.018606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018575 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 10 21:21:02.018606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018590 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 21:21:02.018691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018607 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 21:21:02.018691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018620 2561 kubelet.go:398] "Adding apiserver pod source" Jul 10 21:21:02.018691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.018634 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 21:21:02.019035 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.019020 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 21:21:02.019131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.019044 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 21:21:02.022941 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.022895 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 21:21:02.024594 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.024576 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 21:21:02.024690 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.024608 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 21:21:02.025285 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025274 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 21:21:02.025319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025291 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 21:21:02.025319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025306 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 21:21:02.025319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025312 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 21:21:02.025319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025320 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 21:21:02.025451 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025326 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 21:21:02.025451 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025443 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 21:21:02.025451 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025449 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 21:21:02.025528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025456 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 21:21:02.025528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025461 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 21:21:02.025528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025470 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 21:21:02.025528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.025478 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 21:21:02.026215 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.026206 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 21:21:02.026215 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.026214 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 21:21:02.029667 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.029651 2561 server.go:1268] "Started kubelet" Jul 10 21:21:02.029780 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.029737 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 21:21:02.029870 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.029764 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 21:21:02.029870 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.029837 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 21:21:02.030222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.030180 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 21:21:02.030312 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.030220 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-213.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 21:21:02.030312 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.030289 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 21:21:02.030699 ip-10-0-130-213 systemd[1]: Started Kubernetes Kubelet. Jul 10 21:21:02.031235 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.031124 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 21:21:02.032161 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.032145 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 10 21:21:02.035805 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.035783 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 21:21:02.036886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.036866 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 21:21:02.036886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.036879 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 21:21:02.037026 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.036961 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.037397 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.036047 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-213.ec2.internal.18c10a7d9c94e3bc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-213.ec2.internal,UID:ip-10-0-130-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-213.ec2.internal,},FirstTimestamp:2026-07-10 21:21:02.029620156 +0000 UTC m=+0.380143724,LastTimestamp:2026-07-10 21:21:02.029620156 +0000 UTC m=+0.380143724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-213.ec2.internal,}" Jul 10 21:21:02.037622 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.037575 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 21:21:02.037622 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.037593 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 21:21:02.037822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.037649 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 21:21:02.037822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.037734 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 10 21:21:02.037822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.037743 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 10 21:21:02.038076 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038048 2561 factory.go:153] Registering CRI-O factory Jul 10 21:21:02.038136 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038081 2561 factory.go:223] Registration of the crio container factory successfully Jul 10 21:21:02.038169 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038144 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 21:21:02.038169 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038152 2561 factory.go:55] Registering systemd factory Jul 10 21:21:02.038169 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038157 2561 factory.go:223] Registration of the systemd container factory successfully Jul 10 21:21:02.038273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038174 2561 factory.go:103] Registering Raw factory Jul 10 21:21:02.038273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038187 2561 manager.go:1196] Started watching for new ooms in manager Jul 10 21:21:02.038814 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.038800 2561 manager.go:349] Starting recovery of all containers Jul 10 21:21:02.044794 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.044760 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 21:21:02.044908 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.044860 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 21:21:02.049752 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.049553 2561 manager.go:354] Recovery completed Jul 10 21:21:02.055471 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.055455 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.058421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.057911 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.058421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.057967 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.058421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.057982 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.059295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059146 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 21:21:02.059295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059163 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 21:21:02.059295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059174 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.059295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059184 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 10 21:21:02.059947 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059929 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.060017 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059966 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.060017 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.059980 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.060434 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.060333 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-213.ec2.internal.18c10a7d9e45313e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-213.ec2.internal,UID:ip-10-0-130-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-213.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-213.ec2.internal,},FirstTimestamp:2026-07-10 21:21:02.05795155 +0000 UTC m=+0.408475121,LastTimestamp:2026-07-10 21:21:02.05795155 +0000 UTC m=+0.408475121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-213.ec2.internal,}" Jul 10 21:21:02.060697 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.060684 2561 policy_none.go:49] "None policy: Start" Jul 10 21:21:02.060756 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.060705 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 21:21:02.060756 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.060717 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 21:21:02.061811 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.061800 2561 policy_none.go:47] "Start" Jul 10 21:21:02.066868 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.066848 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgfh4" Jul 10 21:21:02.069824 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.069744 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-213.ec2.internal.18c10a7d9e4588e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-213.ec2.internal,UID:ip-10-0-130-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-213.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-213.ec2.internal,},FirstTimestamp:2026-07-10 21:21:02.057973989 +0000 UTC m=+0.408497560,LastTimestamp:2026-07-10 21:21:02.057973989 +0000 UTC m=+0.408497560,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-213.ec2.internal,}" Jul 10 21:21:02.072676 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.072657 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgfh4" Jul 10 21:21:02.098770 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.098735 2561 manager.go:337] "Starting Device Plugin manager" Jul 10 21:21:02.098770 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.098774 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 21:21:02.099010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.098786 2561 server.go:87] "Starting device plugin registration server" Jul 10 21:21:02.099135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.099122 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 21:21:02.099180 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.099139 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 21:21:02.099249 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.099229 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 21:21:02.099338 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.099320 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 21:21:02.099338 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.099336 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 21:21:02.099970 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.099943 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 21:21:02.100032 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.100016 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.176015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.175976 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 21:21:02.177414 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.177392 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 21:21:02.177470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.177457 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 21:21:02.177515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.177489 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 21:21:02.177556 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.177530 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 21:21:02.180395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.180370 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:02.199953 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.199925 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.203128 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.203108 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.203194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.203148 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.203194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.203160 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.203194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.203186 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.213313 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.213260 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.213313 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.213290 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-213.ec2.internal\": node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.240191 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.240166 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.277884 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.277829 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal"] Jul 10 21:21:02.278011 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.277937 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.278921 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.278904 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.279000 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.278935 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.279000 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.278944 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.280576 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.280563 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.280689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.280674 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.280724 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.280708 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.281348 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281319 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.281465 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281354 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.281465 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281372 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.281465 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281396 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.281465 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281415 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.281465 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.281424 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.282763 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.282748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.282822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.282777 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:02.283487 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.283473 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:02.283570 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.283499 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:02.283570 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.283509 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:02.322236 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.322206 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-213.ec2.internal\" not found" node="ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.327282 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.327258 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-213.ec2.internal\" not found" node="ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.339515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.339477 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8ec75a18410aaf8a2d5adb81c2eec0a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-213.ec2.internal\" (UID: \"c8ec75a18410aaf8a2d5adb81c2eec0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.339515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.339516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.339665 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.339536 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.340607 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.340587 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.439997 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.439940 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.439997 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.440007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8ec75a18410aaf8a2d5adb81c2eec0a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-213.ec2.internal\" (UID: \"c8ec75a18410aaf8a2d5adb81c2eec0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.440269 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.440037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.440269 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.440044 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.440269 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.440069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c889ca818034cea8a0b77f26b61708d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal\" (UID: \"c889ca818034cea8a0b77f26b61708d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.440269 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.440145 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8ec75a18410aaf8a2d5adb81c2eec0a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-213.ec2.internal\" (UID: \"c8ec75a18410aaf8a2d5adb81c2eec0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.441093 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.441056 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.541367 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.541272 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.631411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.631378 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.631507 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.631380 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:02.642222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.642194 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.743025 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.742969 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.843636 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.843536 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.914355 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.914323 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:02.943919 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:02.943878 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:02.985940 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.985893 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 21:21:02.986620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.986070 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 21:21:02.986620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:02.986092 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 21:21:03.044181 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:03.044139 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:03.076952 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.076899 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 21:16:02 +0000 UTC" deadline="2028-01-25 00:16:20.429560831 +0000 UTC" Jul 10 21:21:03.076952 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.076946 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13514h55m17.352618263s" Jul 10 21:21:03.127405 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.127375 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 21:21:03.128909 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.128885 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 21:21:03.129030 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.128923 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 21:21:03.129030 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.128941 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 21:21:03.144389 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:03.144353 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-213.ec2.internal\" not found" Jul 10 21:21:03.213191 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.213154 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:03.220391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.220362 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:03.232620 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:03.232574 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ec75a18410aaf8a2d5adb81c2eec0a.slice/crio-2bc87e7d742bff00b1eefc9095a8a6086fd757688bdf84da8b1859f1dc8529e9 WatchSource:0}: Error finding container 2bc87e7d742bff00b1eefc9095a8a6086fd757688bdf84da8b1859f1dc8529e9: Status 404 returned error can't find the container with id 2bc87e7d742bff00b1eefc9095a8a6086fd757688bdf84da8b1859f1dc8529e9 Jul 10 21:21:03.232996 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:03.232971 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc889ca818034cea8a0b77f26b61708d5.slice/crio-41fd638de6ab4a3d06d3b996265d166510d0e8056f951b5bd18ad42f8818d75d WatchSource:0}: Error finding container 41fd638de6ab4a3d06d3b996265d166510d0e8056f951b5bd18ad42f8818d75d: Status 404 returned error can't find the container with id 41fd638de6ab4a3d06d3b996265d166510d0e8056f951b5bd18ad42f8818d75d Jul 10 21:21:03.237197 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.237168 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" Jul 10 21:21:03.238893 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.238875 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:21:03.248062 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.248041 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 21:21:03.249291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.249275 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" Jul 10 21:21:03.264715 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:03.264685 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 21:21:04.021174 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.021131 2561 apiserver.go:52] "Watching apiserver" Jul 10 21:21:04.031878 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.031841 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 21:21:04.034658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.034629 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-989qr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch","openshift-cluster-node-tuning-operator/tuned-66hdv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal","openshift-multus/multus-additional-cni-plugins-2zczs","openshift-multus/multus-n5hrs","openshift-network-operator/iptables-alerter-lr64f","kube-system/konnectivity-agent-rcbgx","kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal","openshift-dns/node-resolver-649j8","openshift-image-registry/node-ca-cq2g5","openshift-multus/network-metrics-daemon-vfl8r","openshift-network-diagnostics/network-check-target-8gqlw","openshift-ovn-kubernetes/ovnkube-node-t9277"] Jul 10 21:21:04.036795 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.036771 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.037157 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.037128 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 21:21:04.038204 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.038179 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.039753 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.039728 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 21:21:04.039864 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.039737 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 21:21:04.039864 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.039736 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pnnkk\"" Jul 10 21:21:04.040072 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.040051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:04.040174 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.040145 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:04.040569 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.040549 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5dqhx\"" Jul 10 21:21:04.041305 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.041283 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 21:21:04.041926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.041909 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.042163 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.042127 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.042405 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.042385 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 21:21:04.042604 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.042588 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 21:21:04.042821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.042802 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 21:21:04.043572 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.043548 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.043671 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.043644 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.044821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.044513 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.045915 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.045885 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.048482 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048459 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-slash\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048582 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048497 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-netns\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048582 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048524 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-systemd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048582 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-var-lib-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048582 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-etc-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-kubelet\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-systemd-units\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048712 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-log-socket\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-bin\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.048793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-env-overrides\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovn-node-metrics-cert\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048817 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb78a23-940e-4a6c-83bf-c0dba9254184-agent-certs\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048845 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78a23-940e-4a6c-83bf-c0dba9254184-konnectivity-ca\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048867 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-node-log\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048886 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-config\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75vfp\" (UniqueName: \"kubernetes.io/projected/6087ea90-6a0f-4b14-b524-5808dce10ed5-kube-api-access-75vfp\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-ovn\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.048986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-netd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049014 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049042 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-script-lib\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.049590 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049398 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.049948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049926 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 21:21:04.050028 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049974 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 21:21:04.050028 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.049926 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 21:21:04.050192 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050173 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 21:21:04.050298 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050201 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.050298 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k57g9\"" Jul 10 21:21:04.050298 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050265 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.050298 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050201 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050685 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050760 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050882 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.050962 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.051080 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7v5bn\"" Jul 10 21:21:04.051177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.051180 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsz77\"" Jul 10 21:21:04.051425 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.051258 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-klbsf\"" Jul 10 21:21:04.051425 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.051414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.051505 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.051486 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:04.051914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.051894 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.052131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.052107 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 21:21:04.052230 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.052166 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgjzk\"" Jul 10 21:21:04.052288 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.052270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.052363 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.052337 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:04.052495 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.052475 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.054006 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.053984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.054305 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.054284 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 21:21:04.055696 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.055677 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.057727 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.057703 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.057882 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.057859 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cqfbj\"" Jul 10 21:21:04.057976 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.057711 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.058303 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.058282 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 21:21:04.058403 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.058383 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wbrw2\"" Jul 10 21:21:04.059767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.059530 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 21:21:04.059767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.059567 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 21:21:04.074936 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.074896 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lvcph" Jul 10 21:21:04.084693 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.084666 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lvcph" Jul 10 21:21:04.139866 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.139832 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 21:21:04.149418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149380 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-hosts-file\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-etc-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh96w\" (UniqueName: \"kubernetes.io/projected/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-kube-api-access-lh96w\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-registration-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-cnibin\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-conf\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.149603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-multus\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149608 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjpq\" (UniqueName: \"kubernetes.io/projected/5014c522-e157-46e0-9fab-5df447a1d745-kube-api-access-qfjpq\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-config\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75vfp\" (UniqueName: \"kubernetes.io/projected/6087ea90-6a0f-4b14-b524-5808dce10ed5-kube-api-access-75vfp\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149687 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08edcf45-bc59-4eb2-a805-d4a376700481-iptables-alerter-script\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149743 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-ovn\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-script-lib\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-socket-dir-parent\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149825 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-hostroot\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-tmp-dir\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-dbus\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-netns\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.149983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149970 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-modprobe-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.149996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-run\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-sys\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150046 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-host\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-k8s-cni-cncf-io\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-kubelet\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-systemd-units\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-log-socket\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150320 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150388 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-etc-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150401 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-netns\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150443 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-systemd-units\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150445 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-kubelet\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-env-overrides\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150552 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghsc\" (UniqueName: \"kubernetes.io/projected/3ea7267a-368f-438c-810c-fdf71d54bfdb-kube-api-access-rghsc\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150590 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-system-cni-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.150760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-var-lib-kubelet\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150629 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-log-socket\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150654 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-kubelet\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150692 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-multus-certs\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-device-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-systemd\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-tuned\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150842 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.150962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-bin\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151002 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-run-ovn-kubernetes\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151022 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-env-overrides\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151030 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-config\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151041 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151046 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-bin\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.151621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-tmp\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151136 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-daemon-config\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-ovn\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151165 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-etc-kubernetes\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151237 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08edcf45-bc59-4eb2-a805-d4a376700481-host-slash\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovn-node-metrics-cert\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysconfig\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cni-binary-copy\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb78a23-940e-4a6c-83bf-c0dba9254184-agent-certs\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-node-log\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-netns\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151533 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-kubelet-config\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151584 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3229741a-df29-40c4-9abe-18205becc9d0-host\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.152453 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-system-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151660 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-os-release\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151663 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovnkube-script-lib\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151687 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxxp\" (UniqueName: \"kubernetes.io/projected/08edcf45-bc59-4eb2-a805-d4a376700481-kube-api-access-fkxxp\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-slash\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151701 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-node-log\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151910 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-slash\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-systemd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.151980 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-var-lib-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152010 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-lib-modules\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm622\" (UniqueName: \"kubernetes.io/projected/2b94ac16-7614-4b29-89ac-5d6f04d87453-kube-api-access-wm622\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152087 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-run-systemd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152095 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-var-lib-openvswitch\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-os-release\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.153206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152171 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-conf-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152196 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-socket-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-sys-fs\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3229741a-df29-40c4-9abe-18205becc9d0-serviceca\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152285 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7zc\" (UniqueName: \"kubernetes.io/projected/3229741a-df29-40c4-9abe-18205becc9d0-kube-api-access-6v7zc\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78a23-940e-4a6c-83bf-c0dba9254184-konnectivity-ca\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152344 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cnibin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpbg\" (UniqueName: \"kubernetes.io/projected/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-kube-api-access-zwpbg\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-netd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grp4n\" (UniqueName: \"kubernetes.io/projected/7d66904b-9304-4ab3-89e9-042ad744cbb3-kube-api-access-grp4n\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-kubernetes\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-bin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6087ea90-6a0f-4b14-b524-5808dce10ed5-host-cni-netd\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.154018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.152907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78a23-940e-4a6c-83bf-c0dba9254184-konnectivity-ca\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.155933 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.155904 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6087ea90-6a0f-4b14-b524-5808dce10ed5-ovn-node-metrics-cert\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.156120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.156092 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cb78a23-940e-4a6c-83bf-c0dba9254184-agent-certs\") pod \"konnectivity-agent-rcbgx\" (UID: \"3cb78a23-940e-4a6c-83bf-c0dba9254184\") " pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.161053 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.160975 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75vfp\" (UniqueName: \"kubernetes.io/projected/6087ea90-6a0f-4b14-b524-5808dce10ed5-kube-api-access-75vfp\") pod \"ovnkube-node-t9277\" (UID: \"6087ea90-6a0f-4b14-b524-5808dce10ed5\") " pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.183800 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.183726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" event={"ID":"c8ec75a18410aaf8a2d5adb81c2eec0a","Type":"ContainerStarted","Data":"2bc87e7d742bff00b1eefc9095a8a6086fd757688bdf84da8b1859f1dc8529e9"} Jul 10 21:21:04.185660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.185625 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" event={"ID":"c889ca818034cea8a0b77f26b61708d5","Type":"ContainerStarted","Data":"41fd638de6ab4a3d06d3b996265d166510d0e8056f951b5bd18ad42f8818d75d"} Jul 10 21:21:04.252925 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.252883 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh96w\" (UniqueName: \"kubernetes.io/projected/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-kube-api-access-lh96w\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.252925 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.252932 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-registration-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.252950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-cnibin\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.252979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-conf\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-multus\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjpq\" (UniqueName: \"kubernetes.io/projected/5014c522-e157-46e0-9fab-5df447a1d745-kube-api-access-qfjpq\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253138 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-registration-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08edcf45-bc59-4eb2-a805-d4a376700481-iptables-alerter-script\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.253211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253185 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-socket-dir-parent\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-hostroot\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253254 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-tmp-dir\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-multus\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-dbus\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253313 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-conf\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-modprobe-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253373 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-run\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-sys\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253408 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-dbus\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253059 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-cnibin\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253424 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-host\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-k8s-cni-cncf-io\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253495 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghsc\" (UniqueName: \"kubernetes.io/projected/3ea7267a-368f-438c-810c-fdf71d54bfdb-kube-api-access-rghsc\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253519 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-system-cni-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.253525 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-var-lib-kubelet\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.253620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253571 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-modprobe-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.253616 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:04.753571005 +0000 UTC m=+3.104094581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253611 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-var-lib-kubelet\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253640 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-kubelet\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-run\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253717 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-hostroot\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-socket-dir-parent\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253784 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-kubelet\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253799 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-k8s-cni-cncf-io\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-sys\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253869 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-multus-certs\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253874 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-host\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253895 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-device-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253927 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-multus-certs\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-systemd\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.253988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-tuned\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08edcf45-bc59-4eb2-a805-d4a376700481-iptables-alerter-script\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.254384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254018 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-system-cni-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254054 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254077 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254117 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-systemd\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-tmp\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254245 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-device-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254268 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysctl-d\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-daemon-config\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-etc-kubernetes\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254412 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-etc-kubernetes\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254702 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254365 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.254996 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-daemon-config\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08edcf45-bc59-4eb2-a805-d4a376700481-host-slash\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysconfig\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-tmp-dir\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255129 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08edcf45-bc59-4eb2-a805-d4a376700481-host-slash\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.255395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cni-binary-copy\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-sysconfig\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255205 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-netns\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-kubelet-config\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.255288 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3229741a-df29-40c4-9abe-18205becc9d0-host\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.255349 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:04.755329373 +0000 UTC m=+3.105852936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255368 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3229741a-df29-40c4-9abe-18205becc9d0-host\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255407 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-run-netns\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-system-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255439 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-os-release\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255460 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-system-cni-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255466 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxxp\" (UniqueName: \"kubernetes.io/projected/08edcf45-bc59-4eb2-a805-d4a376700481-kube-api-access-fkxxp\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.256297 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-lib-modules\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255533 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm622\" (UniqueName: \"kubernetes.io/projected/2b94ac16-7614-4b29-89ac-5d6f04d87453-kube-api-access-wm622\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255562 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255585 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cni-binary-copy\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-os-release\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-conf-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255646 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-socket-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255660 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64784df-28d5-4bb5-b363-9169dd0308af-kubelet-config\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255726 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-os-release\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255729 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5014c522-e157-46e0-9fab-5df447a1d745-os-release\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255823 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-lib-modules\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-multus-conf-dir\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-sys-fs\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3229741a-df29-40c4-9abe-18205becc9d0-serviceca\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255943 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-socket-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7zc\" (UniqueName: \"kubernetes.io/projected/3229741a-df29-40c4-9abe-18205becc9d0-kube-api-access-6v7zc\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255973 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cnibin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.255978 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-sys-fs\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256001 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpbg\" (UniqueName: \"kubernetes.io/projected/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-kube-api-access-zwpbg\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256030 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grp4n\" (UniqueName: \"kubernetes.io/projected/7d66904b-9304-4ab3-89e9-042ad744cbb3-kube-api-access-grp4n\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256057 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-kubernetes\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256083 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-bin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256110 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-hosts-file\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256562 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256616 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5014c522-e157-46e0-9fab-5df447a1d745-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3229741a-df29-40c4-9abe-18205becc9d0-serviceca\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256912 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-host-var-lib-cni-bin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.256980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-kubernetes\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.257004 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-cnibin\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.257331 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-hosts-file\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.257598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.257377 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b94ac16-7614-4b29-89ac-5d6f04d87453-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.260442 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.260390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-etc-tuned\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.260660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.260640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ea7267a-368f-438c-810c-fdf71d54bfdb-tmp\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.280032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.279924 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghsc\" (UniqueName: \"kubernetes.io/projected/3ea7267a-368f-438c-810c-fdf71d54bfdb-kube-api-access-rghsc\") pod \"tuned-66hdv\" (UID: \"3ea7267a-368f-438c-810c-fdf71d54bfdb\") " pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.285808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.285774 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjpq\" (UniqueName: \"kubernetes.io/projected/5014c522-e157-46e0-9fab-5df447a1d745-kube-api-access-qfjpq\") pod \"multus-additional-cni-plugins-2zczs\" (UID: \"5014c522-e157-46e0-9fab-5df447a1d745\") " pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.289499 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.289475 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:04.289570 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.289503 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:04.289570 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.289514 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:04.289659 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.289591 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:04.789573451 +0000 UTC m=+3.140097022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:04.292042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.292015 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpbg\" (UniqueName: \"kubernetes.io/projected/fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc-kube-api-access-zwpbg\") pod \"node-resolver-649j8\" (UID: \"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc\") " pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.292165 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.292051 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grp4n\" (UniqueName: \"kubernetes.io/projected/7d66904b-9304-4ab3-89e9-042ad744cbb3-kube-api-access-grp4n\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.292937 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.292911 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7zc\" (UniqueName: \"kubernetes.io/projected/3229741a-df29-40c4-9abe-18205becc9d0-kube-api-access-6v7zc\") pod \"node-ca-cq2g5\" (UID: \"3229741a-df29-40c4-9abe-18205becc9d0\") " pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.294852 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.294833 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm622\" (UniqueName: \"kubernetes.io/projected/2b94ac16-7614-4b29-89ac-5d6f04d87453-kube-api-access-wm622\") pod \"aws-ebs-csi-driver-node-8kfch\" (UID: \"2b94ac16-7614-4b29-89ac-5d6f04d87453\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.297702 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.297675 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh96w\" (UniqueName: \"kubernetes.io/projected/8bf67fde-1b45-46f6-9f3e-82da4bf16c1e-kube-api-access-lh96w\") pod \"multus-n5hrs\" (UID: \"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e\") " pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.298867 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.298850 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxxp\" (UniqueName: \"kubernetes.io/projected/08edcf45-bc59-4eb2-a805-d4a376700481-kube-api-access-fkxxp\") pod \"iptables-alerter-lr64f\" (UID: \"08edcf45-bc59-4eb2-a805-d4a376700481\") " pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.352833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.352792 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:04.361989 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.361866 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:04.365633 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:04.365598 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb78a23_940e_4a6c_83bf_c0dba9254184.slice/crio-8037cbc1f919afef66dbebc5766375e73c80cd159ca8313f5ab3eaacde3c577c WatchSource:0}: Error finding container 8037cbc1f919afef66dbebc5766375e73c80cd159ca8313f5ab3eaacde3c577c: Status 404 returned error can't find the container with id 8037cbc1f919afef66dbebc5766375e73c80cd159ca8313f5ab3eaacde3c577c Jul 10 21:21:04.380833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.380637 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cq2g5" Jul 10 21:21:04.380833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.380636 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zczs" Jul 10 21:21:04.383646 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:04.383583 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6087ea90_6a0f_4b14_b524_5808dce10ed5.slice/crio-d7d83956d93c6057706dcde864382bae6c9ca12b35d98dcbefa851d5495852e4 WatchSource:0}: Error finding container d7d83956d93c6057706dcde864382bae6c9ca12b35d98dcbefa851d5495852e4: Status 404 returned error can't find the container with id d7d83956d93c6057706dcde864382bae6c9ca12b35d98dcbefa851d5495852e4 Jul 10 21:21:04.390963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.390932 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5hrs" Jul 10 21:21:04.398078 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.398046 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-649j8" Jul 10 21:21:04.415085 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.415028 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lr64f" Jul 10 21:21:04.424533 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.424499 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66hdv" Jul 10 21:21:04.431934 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.431899 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" Jul 10 21:21:04.578280 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.578181 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:04.760833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.760795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:04.760833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.760850 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:04.761059 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.760972 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:04.761059 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.760981 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:04.761059 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.761044 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:05.761028326 +0000 UTC m=+4.111551883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:04.761059 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.761060 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:05.761052356 +0000 UTC m=+4.111575914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:04.861828 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:04.861784 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:04.862032 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.862005 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:04.862108 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.862046 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:04.862108 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.862061 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:04.862200 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:04.862144 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:05.862123766 +0000 UTC m=+4.212647339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:05.087704 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.087650 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 21:16:04 +0000 UTC" deadline="2027-12-07 02:50:51.234192653 +0000 UTC" Jul 10 21:21:05.087704 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.087700 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12341h29m46.146499561s" Jul 10 21:21:05.188501 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.188409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"d7d83956d93c6057706dcde864382bae6c9ca12b35d98dcbefa851d5495852e4"} Jul 10 21:21:05.189470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.189444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcbgx" event={"ID":"3cb78a23-940e-4a6c-83bf-c0dba9254184","Type":"ContainerStarted","Data":"8037cbc1f919afef66dbebc5766375e73c80cd159ca8313f5ab3eaacde3c577c"} Jul 10 21:21:05.587645 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:05.587601 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf67fde_1b45_46f6_9f3e_82da4bf16c1e.slice/crio-fb9bfb5914e583751c1531c811ace464bcbed47abec7b64611d2c5e2abb70965 WatchSource:0}: Error finding container fb9bfb5914e583751c1531c811ace464bcbed47abec7b64611d2c5e2abb70965: Status 404 returned error can't find the container with id fb9bfb5914e583751c1531c811ace464bcbed47abec7b64611d2c5e2abb70965 Jul 10 21:21:05.605752 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:05.605706 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3229741a_df29_40c4_9abe_18205becc9d0.slice/crio-03b6f2416095ec97159be4b436047ae04e0b5b74b2a7ab35470abf1ec729c904 WatchSource:0}: Error finding container 03b6f2416095ec97159be4b436047ae04e0b5b74b2a7ab35470abf1ec729c904: Status 404 returned error can't find the container with id 03b6f2416095ec97159be4b436047ae04e0b5b74b2a7ab35470abf1ec729c904 Jul 10 21:21:05.607092 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:05.606950 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea7267a_368f_438c_810c_fdf71d54bfdb.slice/crio-f78b1fd09bc13529a311571a9ff701a2f3da9ba69baa0743bc9d1dc9ebdb2af0 WatchSource:0}: Error finding container f78b1fd09bc13529a311571a9ff701a2f3da9ba69baa0743bc9d1dc9ebdb2af0: Status 404 returned error can't find the container with id f78b1fd09bc13529a311571a9ff701a2f3da9ba69baa0743bc9d1dc9ebdb2af0 Jul 10 21:21:05.608017 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:05.607991 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8123ab_1603_45d5_aeef_f5ccf4f8c2bc.slice/crio-db6cce5004df1095bbad6188006a709d251a55de14641f80882e11e1b682ba67 WatchSource:0}: Error finding container db6cce5004df1095bbad6188006a709d251a55de14641f80882e11e1b682ba67: Status 404 returned error can't find the container with id db6cce5004df1095bbad6188006a709d251a55de14641f80882e11e1b682ba67 Jul 10 21:21:05.770912 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.770687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:05.771132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.770956 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:05.771132 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.770872 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:05.771132 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.771093 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:05.771132 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.771105 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:07.77105717 +0000 UTC m=+6.121580753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:05.771293 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.771163 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:07.771148162 +0000 UTC m=+6.121671720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:05.871487 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.871449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:05.871682 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.871656 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:05.871728 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.871692 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:05.871728 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.871709 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:05.871830 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:05.871778 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:07.871755194 +0000 UTC m=+6.222278771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:05.894977 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:05.894944 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 21:21:06.088741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.088624 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 21:16:04 +0000 UTC" deadline="2028-03-03 18:45:01.148559938 +0000 UTC" Jul 10 21:21:06.088741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.088702 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14445h23m55.059862065s" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.180854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:06.180997 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.181465 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:06.181577 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.181647 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:06.181843 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:06.181716 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:06.201283 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.201175 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66hdv" event={"ID":"3ea7267a-368f-438c-810c-fdf71d54bfdb","Type":"ContainerStarted","Data":"f78b1fd09bc13529a311571a9ff701a2f3da9ba69baa0743bc9d1dc9ebdb2af0"} Jul 10 21:21:06.205803 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.205757 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5hrs" event={"ID":"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e","Type":"ContainerStarted","Data":"fb9bfb5914e583751c1531c811ace464bcbed47abec7b64611d2c5e2abb70965"} Jul 10 21:21:06.226161 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.221066 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" event={"ID":"c8ec75a18410aaf8a2d5adb81c2eec0a","Type":"ContainerStarted","Data":"e4ea4c02ff7891a2e4978cea014a97cd6b5c091d598523e2e171f5887973fc95"} Jul 10 21:21:06.229074 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.229024 2561 generic.go:356] "Generic (PLEG): container finished" podID="c889ca818034cea8a0b77f26b61708d5" containerID="c80ac5afc482a4eebc19125f2d4fa2bb9b517ad9838a75cb565a2da700cc2440" exitCode=0 Jul 10 21:21:06.229254 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.229155 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" event={"ID":"c889ca818034cea8a0b77f26b61708d5","Type":"ContainerDied","Data":"c80ac5afc482a4eebc19125f2d4fa2bb9b517ad9838a75cb565a2da700cc2440"} Jul 10 21:21:06.244729 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.244686 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" event={"ID":"2b94ac16-7614-4b29-89ac-5d6f04d87453","Type":"ContainerStarted","Data":"355fe9e66973e61074335f6983e1ffcfc9d45c9381ab8285ee17edbb37b6ec9c"} Jul 10 21:21:06.245822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.245763 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-213.ec2.internal" podStartSLOduration=3.245740235 podStartE2EDuration="3.245740235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:06.244332157 +0000 UTC m=+4.594855759" watchObservedRunningTime="2026-07-10 21:21:06.245740235 +0000 UTC m=+4.596263818" Jul 10 21:21:06.258556 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.258511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr64f" event={"ID":"08edcf45-bc59-4eb2-a805-d4a376700481","Type":"ContainerStarted","Data":"cac35ee86c7e175cbe6ee7f53d137e1167a4c00a085993d552110331d2b54713"} Jul 10 21:21:06.279489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.279395 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-649j8" event={"ID":"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc","Type":"ContainerStarted","Data":"db6cce5004df1095bbad6188006a709d251a55de14641f80882e11e1b682ba67"} Jul 10 21:21:06.281743 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.281646 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cq2g5" event={"ID":"3229741a-df29-40c4-9abe-18205becc9d0","Type":"ContainerStarted","Data":"03b6f2416095ec97159be4b436047ae04e0b5b74b2a7ab35470abf1ec729c904"} Jul 10 21:21:06.286125 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:06.286037 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerStarted","Data":"6ea860019557bda2d3808348d7e0cde8704f48f91ee896a943f7d9cb5f43e290"} Jul 10 21:21:07.313773 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:07.313611 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" event={"ID":"c889ca818034cea8a0b77f26b61708d5","Type":"ContainerStarted","Data":"97c5c8aee68aa63b38a2c1f10cf16b6119fc78a1a8075c28d025b77bbec5045b"} Jul 10 21:21:07.331907 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:07.330765 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-213.ec2.internal" podStartSLOduration=4.330741588 podStartE2EDuration="4.330741588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:07.330519724 +0000 UTC m=+5.681043315" watchObservedRunningTime="2026-07-10 21:21:07.330741588 +0000 UTC m=+5.681265170" Jul 10 21:21:07.788410 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:07.788364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:07.788621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:07.788452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:07.788621 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.788613 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:07.788733 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.788681 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:11.788660422 +0000 UTC m=+10.139184005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:07.788794 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.788733 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:07.788794 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.788769 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:11.788757241 +0000 UTC m=+10.139280799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:07.889619 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:07.889564 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:07.889828 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.889797 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:07.889828 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.889825 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:07.889944 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.889839 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:07.889944 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:07.889912 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:11.88988867 +0000 UTC m=+10.240412229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:08.180853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:08.180812 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:08.181067 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:08.180965 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:08.181444 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:08.181418 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:08.181562 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:08.181537 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:08.181643 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:08.181605 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:08.181727 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:08.181691 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:10.179195 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:10.179163 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:10.179665 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:10.179259 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:10.179665 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:10.179570 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:10.179665 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:10.179630 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:10.179833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:10.179777 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:10.179882 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:10.179838 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:11.825532 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:11.825606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.825728 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.825794 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:19.825774374 +0000 UTC m=+18.176297946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.825862 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:11.825964 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.825925 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:19.825898184 +0000 UTC m=+18.176421744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:11.925984 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:11.925935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:11.926172 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.926104 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:11.926172 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.926127 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:11.926172 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.926136 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:11.926288 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:11.926194 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:19.926178274 +0000 UTC m=+18.276701832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:12.178767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:12.178728 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:12.178925 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:12.178863 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:12.179315 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:12.179286 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:12.179425 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:12.179406 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:12.179592 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:12.179560 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:12.179707 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:12.179651 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:14.178470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:14.178399 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:14.178470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:14.178444 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:14.179897 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:14.179856 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:14.180421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:14.178399 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:14.180663 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:14.180389 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:14.183637 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:14.183605 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:16.181583 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:16.181550 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:16.181583 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:16.181565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:16.181583 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:16.181550 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:16.182055 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:16.181669 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:16.182055 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:16.181704 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:16.182055 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:16.181800 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:18.182017 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:18.181983 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:18.182483 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:18.182029 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:18.182483 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:18.182134 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:18.182483 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:18.182146 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:18.182483 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:18.182201 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:18.182483 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:18.182298 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:19.896718 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:19.896672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:19.896718 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:19.896728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:19.897222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.896831 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:19.897222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.896865 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:19.897222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.896887 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:35.896872501 +0000 UTC m=+34.247396059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:19.897222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.896934 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:21:35.896915615 +0000 UTC m=+34.247439185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:19.997624 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:19.997577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:19.997813 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.997772 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:19.997813 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.997795 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:19.997813 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.997807 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:19.997986 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:19.997865 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:21:35.997846388 +0000 UTC m=+34.348369949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:20.178728 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:20.178651 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:20.178888 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:20.178658 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:20.178888 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:20.178872 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:20.178975 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:20.178769 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:20.178975 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:20.178658 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:20.178975 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:20.178958 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:22.178404 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:22.178356 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:22.178923 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:22.178497 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:22.178923 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:22.178508 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:22.178923 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:22.178528 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:22.178923 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:22.178582 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:22.178923 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:22.178642 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:24.180744 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:24.180714 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:24.181131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:24.180846 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:24.181131 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:24.180851 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:24.181131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:24.180874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:24.181131 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:24.180949 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:24.181131 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:24.181015 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:25.352515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.352153 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="443c4b155a55ac2340021070a28cfb399aeb67b6bed30eee289b2d3cce0c50b4" exitCode=0 Jul 10 21:21:25.353332 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.352240 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"443c4b155a55ac2340021070a28cfb399aeb67b6bed30eee289b2d3cce0c50b4"} Jul 10 21:21:25.354295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.354268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66hdv" event={"ID":"3ea7267a-368f-438c-810c-fdf71d54bfdb","Type":"ContainerStarted","Data":"01cfc96674b2add5435c1055d0c12ea0d3b7d89bfd4f2d75b875fc6542480d83"} Jul 10 21:21:25.356132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.356102 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5hrs" event={"ID":"8bf67fde-1b45-46f6-9f3e-82da4bf16c1e","Type":"ContainerStarted","Data":"b028cb580ca26386840462b8771e4b3023e79e4fe176dc277b7f3ff2cf290d90"} Jul 10 21:21:25.359096 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359054 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:21:25.359451 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359429 2561 generic.go:356] "Generic (PLEG): container finished" podID="6087ea90-6a0f-4b14-b524-5808dce10ed5" containerID="d21ff80904472b04c8f5a43cc0a9deb8c941c7413ca2d6210daee29ed5c2a95d" exitCode=1 Jul 10 21:21:25.359517 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359497 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"3ac302c75b96c6db1e09d9f4ffa4ba682d89656f8fc2ad22fb31702eba7d2b9a"} Jul 10 21:21:25.359598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"dc50571a293966a3e02376cc1deabc89512af2a951a5bb715e1401aabb01b4ef"} Jul 10 21:21:25.359656 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"d1bdd0b27f060d2d11d3869e2efdc489efa74615aab81c6db83efc19dedd74a8"} Jul 10 21:21:25.359714 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"6ed18e46b317c154f980b3be8300bbfcedfb723c52d7af55f5420996d8ef7186"} Jul 10 21:21:25.359766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359741 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerDied","Data":"d21ff80904472b04c8f5a43cc0a9deb8c941c7413ca2d6210daee29ed5c2a95d"} Jul 10 21:21:25.359814 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.359792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"e9c52746861dc895ff12d7db782406b975868c628f851dc06db8a12bd6af41fd"} Jul 10 21:21:25.361037 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.361010 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" event={"ID":"2b94ac16-7614-4b29-89ac-5d6f04d87453","Type":"ContainerStarted","Data":"b8b718fd2e2229f3ef48ec9fc3b0d6818b705e0b919c68f8cd83541943584ca3"} Jul 10 21:21:25.362454 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.362428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-649j8" event={"ID":"fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc","Type":"ContainerStarted","Data":"866ec16f2133dd6b0f62de2fae1a394a4ec1fbddab69c6d67dc4383380218d8e"} Jul 10 21:21:25.363717 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.363689 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cq2g5" event={"ID":"3229741a-df29-40c4-9abe-18205becc9d0","Type":"ContainerStarted","Data":"f6952b79eda5b85b35404ad1d9db31217699d732edc6ce99d40c01b4094729ac"} Jul 10 21:21:25.364976 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.364957 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcbgx" event={"ID":"3cb78a23-940e-4a6c-83bf-c0dba9254184","Type":"ContainerStarted","Data":"95cc7cb28d603bf38f761619dd433fc5b99906edc309c51ad187a4a9df838b98"} Jul 10 21:21:25.394380 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.394330 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cq2g5" podStartSLOduration=4.727310865 podStartE2EDuration="23.394311473s" totalImagesPullingTime="18.667000608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:25.394083045 +0000 UTC m=+23.744606635" watchObservedRunningTime="2026-07-10 21:21:25.394311473 +0000 UTC m=+23.744835108" Jul 10 21:21:25.409934 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.409872 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-649j8" podStartSLOduration=4.745397689 podStartE2EDuration="23.409852586s" totalImagesPullingTime="18.664454897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:25.409696216 +0000 UTC m=+23.760219805" watchObservedRunningTime="2026-07-10 21:21:25.409852586 +0000 UTC m=+23.760376166" Jul 10 21:21:25.431822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.431768 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rcbgx" podStartSLOduration=11.81420612 podStartE2EDuration="23.431746153s" totalImagesPullingTime="11.617540033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:25.431120904 +0000 UTC m=+23.781644483" watchObservedRunningTime="2026-07-10 21:21:25.431746153 +0000 UTC m=+23.782269732" Jul 10 21:21:25.450296 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.450240 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n5hrs" podStartSLOduration=4.745140961 podStartE2EDuration="23.450221061s" totalImagesPullingTime="18.7050801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:25.449728583 +0000 UTC m=+23.800252160" watchObservedRunningTime="2026-07-10 21:21:25.450221061 +0000 UTC m=+23.800744649" Jul 10 21:21:25.466640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.466588 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-66hdv" podStartSLOduration=3.79831871 podStartE2EDuration="22.466568s" totalImagesPullingTime="18.66824929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:25.466523832 +0000 UTC m=+23.817047431" watchObservedRunningTime="2026-07-10 21:21:25.466568 +0000 UTC m=+23.817091581" Jul 10 21:21:25.662974 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:25.662950 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 21:21:26.111827 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.111666 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T21:21:25.662970348Z","UUID":"78624d1c-973c-4d5a-8b07-b235c98a8344","Handler":null,"Name":"","Endpoint":""} Jul 10 21:21:26.113735 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.113710 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 21:21:26.113735 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.113750 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 21:21:26.177876 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.177836 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:26.178042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.177852 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:26.178042 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:26.177988 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:26.178042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.177852 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:26.178179 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:26.178035 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:26.178179 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:26.178121 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:26.370050 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.369960 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" event={"ID":"2b94ac16-7614-4b29-89ac-5d6f04d87453","Type":"ContainerStarted","Data":"f068e851c52d9a80ed724ca7c2851533985648433b824a70077c524cc9c60a74"} Jul 10 21:21:26.372213 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.372163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lr64f" event={"ID":"08edcf45-bc59-4eb2-a805-d4a376700481","Type":"ContainerStarted","Data":"cdfeda11740ef36090defa90293531bf5ea71185030867ca3dc97712c2fd2932"} Jul 10 21:21:26.388083 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:26.388008 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lr64f" podStartSLOduration=4.7289233809999995 podStartE2EDuration="23.387985525s" totalImagesPullingTime="18.659062144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:26.387510118 +0000 UTC m=+24.738033696" watchObservedRunningTime="2026-07-10 21:21:26.387985525 +0000 UTC m=+24.738509104" Jul 10 21:21:27.378056 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:27.378027 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:21:27.378532 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:27.378482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"498c388b7754d0b7c4c9510a2307b121c7a9d8cf4fcb732d78356a6e135c79a1"} Jul 10 21:21:27.380889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:27.380860 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" event={"ID":"2b94ac16-7614-4b29-89ac-5d6f04d87453","Type":"ContainerStarted","Data":"fbefb5c19038669e7004d3a56f470ebbc0adfc5887119644ff51260b09ea27bd"} Jul 10 21:21:27.401653 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:27.401591 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kfch" podStartSLOduration=3.771259705 podStartE2EDuration="24.401573057s" totalImagesPullingTime="20.630313352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:27.400751228 +0000 UTC m=+25.751274807" watchObservedRunningTime="2026-07-10 21:21:27.401573057 +0000 UTC m=+25.752096612" Jul 10 21:21:28.177899 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:28.177712 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:28.178134 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:28.177712 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:28.178134 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:28.177970 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:28.178134 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:28.178077 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:28.178134 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:28.177739 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:28.178311 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:28.178183 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:28.378175 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:28.378137 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:28.378850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:28.378830 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:30.178803 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:30.178763 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:30.179490 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:30.178759 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:30.179490 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:30.178901 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:30.179490 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:30.178922 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:30.179490 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:30.178999 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:30.179490 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:30.179076 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:31.394242 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.394044 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:21:31.394860 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.394579 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"c888dadeec9a33d0b0adab7a40d3418696c62b87c01b6fc843cad2f52577c4bd"} Jul 10 21:21:31.394860 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.394798 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:31.394994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.394890 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:31.395075 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.395039 2561 scope.go:117] "RemoveContainer" containerID="d21ff80904472b04c8f5a43cc0a9deb8c941c7413ca2d6210daee29ed5c2a95d" Jul 10 21:21:31.398858 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.398829 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="596eb5d0a846e4fdd426c64a0c4ea2e34ea532becaf23e21c1a1d1ab7570c2f4" exitCode=0 Jul 10 21:21:31.398957 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.398879 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"596eb5d0a846e4fdd426c64a0c4ea2e34ea532becaf23e21c1a1d1ab7570c2f4"} Jul 10 21:21:31.412903 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.412873 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:31.761963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.761860 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:31.762276 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.762056 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 21:21:31.762683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:31.762660 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rcbgx" Jul 10 21:21:32.179804 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.179768 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:32.179989 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.179881 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:32.179989 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.179919 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:32.179989 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.179920 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:32.180155 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.179997 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:32.180155 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.180102 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:32.405120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.405084 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:21:32.405525 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.405460 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" event={"ID":"6087ea90-6a0f-4b14-b524-5808dce10ed5","Type":"ContainerStarted","Data":"8eeba9ac0e66784cdb719e617107b86f2876956ab3877e48ad8dca60b26f5bf1"} Jul 10 21:21:32.405872 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.405847 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:32.422975 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.422947 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:21:32.437461 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.437366 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" podStartSLOduration=10.48702922 podStartE2EDuration="30.437350546s" totalImagesPullingTime="19.950321326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:32.437089378 +0000 UTC m=+30.787612954" watchObservedRunningTime="2026-07-10 21:21:32.437350546 +0000 UTC m=+30.787874123" Jul 10 21:21:32.776868 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.776780 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-989qr"] Jul 10 21:21:32.777007 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.776905 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:32.777044 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.777004 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:32.780407 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.780382 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gqlw"] Jul 10 21:21:32.780535 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.780467 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:32.780585 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.780542 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:32.785430 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.785403 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfl8r"] Jul 10 21:21:32.785526 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:32.785505 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:32.785637 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:32.785612 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:33.409821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:33.409730 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="f2932045e0a796fa1c280862a3b9300ae13ca82b0faeec168d167bd456756975" exitCode=0 Jul 10 21:21:33.409821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:33.409810 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"f2932045e0a796fa1c280862a3b9300ae13ca82b0faeec168d167bd456756975"} Jul 10 21:21:34.178666 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:34.178489 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:34.178876 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:34.178500 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:34.178876 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:34.178769 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:34.178876 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:34.178504 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:34.178876 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:34.178836 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:34.179110 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:34.178888 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:35.415603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:35.415517 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="1f3a40eaf3b081cf59e95a7f2f4fc329ae1950fce29e7d1f49addb0c55c5415d" exitCode=0 Jul 10 21:21:35.415603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:35.415592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"1f3a40eaf3b081cf59e95a7f2f4fc329ae1950fce29e7d1f49addb0c55c5415d"} Jul 10 21:21:35.905889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:35.905848 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:35.906153 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:35.905922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:35.906153 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:35.906009 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:35.906153 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:35.906009 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:35.906153 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:35.906093 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret podName:b64784df-28d5-4bb5-b363-9169dd0308af nodeName:}" failed. No retries permitted until 2026-07-10 21:22:07.906058208 +0000 UTC m=+66.256581767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret") pod "global-pull-secret-syncer-989qr" (UID: "b64784df-28d5-4bb5-b363-9169dd0308af") : object "kube-system"/"original-pull-secret" not registered Jul 10 21:21:35.906153 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:35.906114 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs podName:7d66904b-9304-4ab3-89e9-042ad744cbb3 nodeName:}" failed. No retries permitted until 2026-07-10 21:22:07.906103374 +0000 UTC m=+66.256626939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs") pod "network-metrics-daemon-vfl8r" (UID: "7d66904b-9304-4ab3-89e9-042ad744cbb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 21:21:36.006657 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.006622 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:36.006840 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.006807 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 21:21:36.006840 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.006831 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 21:21:36.006919 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.006847 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qbb2z for pod openshift-network-diagnostics/network-check-target-8gqlw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:36.006919 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.006901 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z podName:c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.006886948 +0000 UTC m=+66.357410504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbb2z" (UniqueName: "kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z") pod "network-check-target-8gqlw" (UID: "c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 21:21:36.181792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.181701 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:36.181792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.181776 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:36.181975 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.181791 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:36.181975 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.181890 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-989qr" podUID="b64784df-28d5-4bb5-b363-9169dd0308af" Jul 10 21:21:36.182094 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.182032 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfl8r" podUID="7d66904b-9304-4ab3-89e9-042ad744cbb3" Jul 10 21:21:36.182155 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:36.182122 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gqlw" podUID="c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a" Jul 10 21:21:36.936111 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.936080 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-213.ec2.internal" event="NodeReady" Jul 10 21:21:36.936679 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.936241 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 21:21:36.978717 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.978681 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:21:36.989528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.989485 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww"] Jul 10 21:21:36.989719 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.989638 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:36.992940 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.992916 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 21:21:36.993580 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.993559 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 21:21:36.994160 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.994133 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lkx9k\"" Jul 10 21:21:36.997278 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:36.997257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 21:21:37.001606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.001507 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68f44597c6-hzxv6"] Jul 10 21:21:37.002174 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.002152 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.003450 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.003428 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 21:21:37.008463 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.008436 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4nbjf\"" Jul 10 21:21:37.008860 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.008840 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 21:21:37.009179 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.009053 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.009941 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.009915 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 21:21:37.010411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.010389 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.015538 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.015261 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9"] Jul 10 21:21:37.018204 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.017485 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.020893 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.020868 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 21:21:37.022309 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.021992 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 21:21:37.022309 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.022010 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.022560 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.022483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 21:21:37.022560 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.022534 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q5rzv\"" Jul 10 21:21:37.022686 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.022566 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.022686 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.022488 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 21:21:37.040799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.040760 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rcbgx_3cb78a23-940e-4a6c-83bf-c0dba9254184/konnectivity-agent/0.log" limit=1024 Jul 10 21:21:37.043325 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.043296 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z"] Jul 10 21:21:37.043468 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.043431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.046306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.045897 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 21:21:37.046306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.046038 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.046306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.046041 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.046306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.046167 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2s4pr\"" Jul 10 21:21:37.046306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.046313 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 21:21:37.061970 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.061936 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd"] Jul 10 21:21:37.062132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.062113 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.064528 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.064502 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 21:21:37.064994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.064747 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 21:21:37.064994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.064766 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.064994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.064843 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.064994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.064761 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rnggk\"" Jul 10 21:21:37.081745 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.081710 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-d642m"] Jul 10 21:21:37.082048 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.082022 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.084615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.084208 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 21:21:37.084615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.084229 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dtwtt\"" Jul 10 21:21:37.084615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.084512 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.084615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.084577 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.103908 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.103823 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qqx7l"] Jul 10 21:21:37.104091 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.103962 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.106656 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.106631 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 21:21:37.106812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.106689 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.106812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.106748 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 21:21:37.106812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.106756 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rdshj\"" Jul 10 21:21:37.106960 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.106890 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.113022 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.112999 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 21:21:37.114859 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9pr\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.114983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.114983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb8d\" (UniqueName: \"kubernetes.io/projected/32d29f1c-89cd-42f6-b9b6-e08b28e47266-kube-api-access-2xb8d\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.114983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114934 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw8h\" (UniqueName: \"kubernetes.io/projected/4fb4b267-19c8-4f61-b649-35dfff7bec04-kube-api-access-tzw8h\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.114983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.114983 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fb4b267-19c8-4f61-b649-35dfff7bec04-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115056 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v8mh\" (UniqueName: \"kubernetes.io/projected/9781f26e-f121-4bdf-a5df-6507501cf548-kube-api-access-9v8mh\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9781f26e-f121-4bdf-a5df-6507501cf548-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-default-certificate\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9781f26e-f121-4bdf-a5df-6507501cf548-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115383 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-stats-auth\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.115550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.115413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.130086 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.130030 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q"] Jul 10 21:21:37.130254 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.130189 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.133371 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.133226 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 21:21:37.133371 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.133246 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.133371 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.133232 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgvps\"" Jul 10 21:21:37.133623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.133557 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.143745 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.143698 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 21:21:37.144787 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144802 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144822 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144841 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144858 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-d642m"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144876 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f44597c6-hzxv6"] Jul 10 21:21:37.144926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144896 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87"] Jul 10 21:21:37.145240 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.144972 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" Jul 10 21:21:37.145512 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.145490 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 21:21:37.147368 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.147347 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.147486 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.147422 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zxfpn\"" Jul 10 21:21:37.147910 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.147888 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.153687 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.153663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rcbgx_3cb78a23-940e-4a6c-83bf-c0dba9254184/konnectivity-agent/0.log" limit=1024 Jul 10 21:21:37.171029 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.170996 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-m5bj4"] Jul 10 21:21:37.171208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.171138 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" Jul 10 21:21:37.173710 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.173667 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.173710 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.173713 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.173951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.173907 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fqmhh\"" Jul 10 21:21:37.186968 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.186939 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:21:37.187146 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.186991 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qqx7l"] Jul 10 21:21:37.187146 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.187015 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5t7db"] Jul 10 21:21:37.187225 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.187149 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.189668 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.189644 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 21:21:37.189920 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.189902 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 21:21:37.190155 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.190140 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ns8ld\"" Jul 10 21:21:37.205129 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.205096 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9x8k5"] Jul 10 21:21:37.205379 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.205334 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.207659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.207637 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jkgdz\"" Jul 10 21:21:37.207802 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.207637 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 21:21:37.207878 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.207850 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 21:21:37.207878 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.207865 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 21:21:37.215909 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.215886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.216036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.215924 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.216036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.215986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9781f26e-f121-4bdf-a5df-6507501cf548-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.216036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216011 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.216036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.216092 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.71604626 +0000 UTC m=+36.066569822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-default-certificate\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216200 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9781f26e-f121-4bdf-a5df-6507501cf548-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216227 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.216274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216257 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddjt\" (UniqueName: \"kubernetes.io/projected/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-kube-api-access-cddjt\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216287 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de2aecbb-9a62-48c4-887b-92b03ac8392b-serving-cert\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9pr\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216355 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb8d\" (UniqueName: \"kubernetes.io/projected/32d29f1c-89cd-42f6-b9b6-e08b28e47266-kube-api-access-2xb8d\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216387 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216412 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-trusted-ca\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-snapshots\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216489 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmd5\" (UniqueName: \"kubernetes.io/projected/28c084d6-5a77-48db-a48a-6bf88d2d91be-kube-api-access-cmmd5\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.216623 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216535 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.217016 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.216697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw8h\" (UniqueName: \"kubernetes.io/projected/4fb4b267-19c8-4f61-b649-35dfff7bec04-kube-api-access-tzw8h\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.217016 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.217002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.217127 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.217035 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-config\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.218607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.217511 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fb4b267-19c8-4f61-b649-35dfff7bec04-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.218607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.217551 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v8mh\" (UniqueName: \"kubernetes.io/projected/9781f26e-f121-4bdf-a5df-6507501cf548-kube-api-access-9v8mh\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.218607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218329 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fb4b267-19c8-4f61-b649-35dfff7bec04-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.218607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218547 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.218607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218612 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9781f26e-f121-4bdf-a5df-6507501cf548-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218633 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218677 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-config\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.218736 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.218751 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d692h\" (UniqueName: \"kubernetes.io/projected/de2aecbb-9a62-48c4-887b-92b03ac8392b-kube-api-access-d692h\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.218805 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.718786843 +0000 UTC m=+36.069310422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218896 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7qw\" (UniqueName: \"kubernetes.io/projected/92294bbd-9727-4430-9c3b-875d7c67e510-kube-api-access-pn7qw\") pod \"volume-data-source-validator-78495597f6-hj44q\" (UID: \"92294bbd-9727-4430-9c3b-875d7c67e510\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.218988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-stats-auth\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.219015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.219007 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-serving-cert\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.219031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-tmp\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.219056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.219099 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzsb\" (UniqueName: \"kubernetes.io/projected/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-kube-api-access-znzsb\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.219210 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.219234 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.219299 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.719283869 +0000 UTC m=+36.069807442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.219367 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:37.219682 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.219402 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.7193918 +0000 UTC m=+36.069915377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:37.221216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.221191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.221742 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.221721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.222318 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.222286 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9781f26e-f121-4bdf-a5df-6507501cf548-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.222812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.222793 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-default-certificate\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.223816 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.223795 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-stats-auth\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.225698 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.225552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rcbgx_3cb78a23-940e-4a6c-83bf-c0dba9254184/konnectivity-agent/0.log" limit=1024 Jul 10 21:21:37.227033 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.227009 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9pr\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.227286 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.227265 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw8h\" (UniqueName: \"kubernetes.io/projected/4fb4b267-19c8-4f61-b649-35dfff7bec04-kube-api-access-tzw8h\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.227515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.227461 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb8d\" (UniqueName: \"kubernetes.io/projected/32d29f1c-89cd-42f6-b9b6-e08b28e47266-kube-api-access-2xb8d\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.227881 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.227856 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v8mh\" (UniqueName: \"kubernetes.io/projected/9781f26e-f121-4bdf-a5df-6507501cf548-kube-api-access-9v8mh\") pod \"kube-storage-version-migrator-operator-799cb7f958-5hgv9\" (UID: \"9781f26e-f121-4bdf-a5df-6507501cf548\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.229416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.229395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.231439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231420 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87"] Jul 10 21:21:37.231571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231451 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-m5bj4"] Jul 10 21:21:37.231571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd"] Jul 10 21:21:37.231571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231506 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t7db"] Jul 10 21:21:37.231571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231531 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x8k5"] Jul 10 21:21:37.231571 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.231564 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.233699 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.233651 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 21:21:37.233834 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.233791 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 21:21:37.233890 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.233856 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hw29l\"" Jul 10 21:21:37.320398 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddjt\" (UniqueName: \"kubernetes.io/projected/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-kube-api-access-cddjt\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.320398 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acb3f997-86fd-499f-9b1b-f9b657fbfdac-config-volume\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de2aecbb-9a62-48c4-887b-92b03ac8392b-serving-cert\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320472 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac6d2700-7161-4eea-a941-329fa08026ac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320501 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2h7\" (UniqueName: \"kubernetes.io/projected/a256bf08-cc43-4cae-ade0-910b4772530d-kube-api-access-ht2h7\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjdt\" (UniqueName: \"kubernetes.io/projected/ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb-kube-api-access-zbjdt\") pod \"network-check-source-869dcfcdcd-ljr87\" (UID: \"ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320580 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-trusted-ca\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-snapshots\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.320658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmd5\" (UniqueName: \"kubernetes.io/projected/28c084d6-5a77-48db-a48a-6bf88d2d91be-kube-api-access-cmmd5\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-config\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a256bf08-cc43-4cae-ade0-910b4772530d-cert\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320775 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-config\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320801 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d692h\" (UniqueName: \"kubernetes.io/projected/de2aecbb-9a62-48c4-887b-92b03ac8392b-kube-api-access-d692h\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7qw\" (UniqueName: \"kubernetes.io/projected/92294bbd-9727-4430-9c3b-875d7c67e510-kube-api-access-pn7qw\") pod \"volume-data-source-validator-78495597f6-hj44q\" (UID: \"92294bbd-9727-4430-9c3b-875d7c67e510\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6snn\" (UniqueName: \"kubernetes.io/projected/acb3f997-86fd-499f-9b1b-f9b657fbfdac-kube-api-access-n6snn\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb3f997-86fd-499f-9b1b-f9b657fbfdac-tmp-dir\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.320995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.320990 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-serving-cert\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.321539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.321539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321057 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-tmp\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.321539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzsb\" (UniqueName: \"kubernetes.io/projected/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-kube-api-access-znzsb\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.321539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.321539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.321928 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.321933 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.322015 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.821996043 +0000 UTC m=+36.172519602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.322432 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2aecbb-9a62-48c4-887b-92b03ac8392b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.322542 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-tmp\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.323009 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de2aecbb-9a62-48c4-887b-92b03ac8392b-snapshots\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.323045 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-config\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.323824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.323781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-config\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.324247 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.324019 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-trusted-ca\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.324247 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.324131 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-serving-cert\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.324489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.324468 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de2aecbb-9a62-48c4-887b-92b03ac8392b-serving-cert\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.326574 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.326554 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-serving-cert\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.329845 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.329817 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7qw\" (UniqueName: \"kubernetes.io/projected/92294bbd-9727-4430-9c3b-875d7c67e510-kube-api-access-pn7qw\") pod \"volume-data-source-validator-78495597f6-hj44q\" (UID: \"92294bbd-9727-4430-9c3b-875d7c67e510\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" Jul 10 21:21:37.330000 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.329857 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzsb\" (UniqueName: \"kubernetes.io/projected/f47a0d11-6bb6-4368-812b-c41aa6b2ccbb-kube-api-access-znzsb\") pod \"console-operator-7866497494-d642m\" (UID: \"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb\") " pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.331146 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.331127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d692h\" (UniqueName: \"kubernetes.io/projected/de2aecbb-9a62-48c4-887b-92b03ac8392b-kube-api-access-d692h\") pod \"insights-operator-68d9dd94cf-qqx7l\" (UID: \"de2aecbb-9a62-48c4-887b-92b03ac8392b\") " pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.331794 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.331765 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddjt\" (UniqueName: \"kubernetes.io/projected/68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d-kube-api-access-cddjt\") pod \"service-ca-operator-7695fc6c8c-nz82z\" (UID: \"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.340151 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.340123 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmd5\" (UniqueName: \"kubernetes.io/projected/28c084d6-5a77-48db-a48a-6bf88d2d91be-kube-api-access-cmmd5\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.355034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.354952 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" Jul 10 21:21:37.373209 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.373173 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" Jul 10 21:21:37.416425 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.416391 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:37.421999 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.421964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a256bf08-cc43-4cae-ade0-910b4772530d-cert\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.421999 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6snn\" (UniqueName: \"kubernetes.io/projected/acb3f997-86fd-499f-9b1b-f9b657fbfdac-kube-api-access-n6snn\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.422251 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422028 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.422251 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb3f997-86fd-499f-9b1b-f9b657fbfdac-tmp-dir\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.422251 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.422251 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422196 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acb3f997-86fd-499f-9b1b-f9b657fbfdac-config-volume\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.422251 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422225 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac6d2700-7161-4eea-a941-329fa08026ac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.422700 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422255 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2h7\" (UniqueName: \"kubernetes.io/projected/a256bf08-cc43-4cae-ade0-910b4772530d-kube-api-access-ht2h7\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.422700 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.422294 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjdt\" (UniqueName: \"kubernetes.io/projected/ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb-kube-api-access-zbjdt\") pod \"network-check-source-869dcfcdcd-ljr87\" (UID: \"ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.423086 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.423180 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.923155507 +0000 UTC m=+36.273679063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.423484 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.423562 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:37.923544319 +0000 UTC m=+36.274067898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.423810 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac6d2700-7161-4eea-a941-329fa08026ac-nginx-conf\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.423944 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb3f997-86fd-499f-9b1b-f9b657fbfdac-tmp-dir\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.423995 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.423954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acb3f997-86fd-499f-9b1b-f9b657fbfdac-config-volume\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.425574 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.425540 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a256bf08-cc43-4cae-ade0-910b4772530d-cert\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.434603 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.434567 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2h7\" (UniqueName: \"kubernetes.io/projected/a256bf08-cc43-4cae-ade0-910b4772530d-kube-api-access-ht2h7\") pod \"ingress-canary-5t7db\" (UID: \"a256bf08-cc43-4cae-ade0-910b4772530d\") " pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.435336 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.435202 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6snn\" (UniqueName: \"kubernetes.io/projected/acb3f997-86fd-499f-9b1b-f9b657fbfdac-kube-api-access-n6snn\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.435471 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.435448 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjdt\" (UniqueName: \"kubernetes.io/projected/ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb-kube-api-access-zbjdt\") pod \"network-check-source-869dcfcdcd-ljr87\" (UID: \"ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" Jul 10 21:21:37.445665 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.445458 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" Jul 10 21:21:37.454924 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.454858 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" Jul 10 21:21:37.482584 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.482307 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" Jul 10 21:21:37.516743 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.516715 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t7db" Jul 10 21:21:37.641104 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.636134 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9"] Jul 10 21:21:37.641104 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.640327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z"] Jul 10 21:21:37.650273 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.649668 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dc5cd2_f5b4_41a7_abae_82f9ec7ce79d.slice/crio-15fce1f0094f29f2ba47e3262da751b34da2d6ff10f33519b8f7b20afa3ab6e7 WatchSource:0}: Error finding container 15fce1f0094f29f2ba47e3262da751b34da2d6ff10f33519b8f7b20afa3ab6e7: Status 404 returned error can't find the container with id 15fce1f0094f29f2ba47e3262da751b34da2d6ff10f33519b8f7b20afa3ab6e7 Jul 10 21:21:37.650273 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.649890 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9781f26e_f121_4bdf_a5df_6507501cf548.slice/crio-dc495075ba5efc5d4a7bd5367d602c5b2d81bc21f1550d84855361aaf8334155 WatchSource:0}: Error finding container dc495075ba5efc5d4a7bd5367d602c5b2d81bc21f1550d84855361aaf8334155: Status 404 returned error can't find the container with id dc495075ba5efc5d4a7bd5367d602c5b2d81bc21f1550d84855361aaf8334155 Jul 10 21:21:37.667989 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.667944 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-d642m"] Jul 10 21:21:37.674351 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.672856 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47a0d11_6bb6_4368_812b_c41aa6b2ccbb.slice/crio-6a7425224138f3dd8db65f0ee624d3a21900044da3b0d9dc16e1cbb4902f94a7 WatchSource:0}: Error finding container 6a7425224138f3dd8db65f0ee624d3a21900044da3b0d9dc16e1cbb4902f94a7: Status 404 returned error can't find the container with id 6a7425224138f3dd8db65f0ee624d3a21900044da3b0d9dc16e1cbb4902f94a7 Jul 10 21:21:37.724347 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.724265 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qqx7l"] Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.728201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.728291 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.728335 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.728365 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728478 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728527 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.728512705 +0000 UTC m=+37.079036261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728868 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.72885729 +0000 UTC m=+37.079380847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728921 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728932 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728955 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.728947993 +0000 UTC m=+37.079471548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.728993 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:37.729053 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.729018 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.729011167 +0000 UTC m=+37.079534725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:37.730224 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.730189 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2aecbb_9a62_48c4_887b_92b03ac8392b.slice/crio-b590972903c3415911d6e6d7b72aa419a52b6413f64b8e7a0a8aa702ebbddb6d WatchSource:0}: Error finding container b590972903c3415911d6e6d7b72aa419a52b6413f64b8e7a0a8aa702ebbddb6d: Status 404 returned error can't find the container with id b590972903c3415911d6e6d7b72aa419a52b6413f64b8e7a0a8aa702ebbddb6d Jul 10 21:21:37.755956 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.755917 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87"] Jul 10 21:21:37.760166 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.760132 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbffab9_4f2b_4ad9_8706_ed656e2ee5cb.slice/crio-d83297777c985d6ee2470aff06c202671d100e43be0da96b26738db1a7963631 WatchSource:0}: Error finding container d83297777c985d6ee2470aff06c202671d100e43be0da96b26738db1a7963631: Status 404 returned error can't find the container with id d83297777c985d6ee2470aff06c202671d100e43be0da96b26738db1a7963631 Jul 10 21:21:37.772176 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.772146 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q"] Jul 10 21:21:37.777167 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.777135 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92294bbd_9727_4430_9c3b_875d7c67e510.slice/crio-d1383da1efbfa118293f67df294df15f3886f3b71428091291910f20bbf5f5d6 WatchSource:0}: Error finding container d1383da1efbfa118293f67df294df15f3886f3b71428091291910f20bbf5f5d6: Status 404 returned error can't find the container with id d1383da1efbfa118293f67df294df15f3886f3b71428091291910f20bbf5f5d6 Jul 10 21:21:37.805576 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.805538 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t7db"] Jul 10 21:21:37.808744 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:37.808711 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda256bf08_cc43_4cae_ade0_910b4772530d.slice/crio-21d55ed178dbaa442beaf242d8ba8dea2a5a71b3c3edff31577c262161bd3674 WatchSource:0}: Error finding container 21d55ed178dbaa442beaf242d8ba8dea2a5a71b3c3edff31577c262161bd3674: Status 404 returned error can't find the container with id 21d55ed178dbaa442beaf242d8ba8dea2a5a71b3c3edff31577c262161bd3674 Jul 10 21:21:37.829520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.829483 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:37.829703 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.829605 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:37.829703 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.829688 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.829668096 +0000 UTC m=+37.180191652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:37.930947 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.930857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:37.930947 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:37.930927 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:37.931192 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.931033 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:37.931192 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.931096 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:37.931192 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.931115 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.931099333 +0000 UTC m=+37.281622888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:37.931192 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:37.931150 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:38.931137846 +0000 UTC m=+37.281661402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:38.178642 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.178605 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:21:38.179244 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.178605 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:21:38.179244 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.178610 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:21:38.181472 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.181222 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 21:21:38.181472 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.181280 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 21:21:38.181472 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.181328 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nklrg\"" Jul 10 21:21:38.181698 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.181620 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j44v2\"" Jul 10 21:21:38.429015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.428974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" event={"ID":"ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb","Type":"ContainerStarted","Data":"d83297777c985d6ee2470aff06c202671d100e43be0da96b26738db1a7963631"} Jul 10 21:21:38.430517 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.430485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" event={"ID":"92294bbd-9727-4430-9c3b-875d7c67e510","Type":"ContainerStarted","Data":"d1383da1efbfa118293f67df294df15f3886f3b71428091291910f20bbf5f5d6"} Jul 10 21:21:38.432164 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.432084 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d642m" event={"ID":"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb","Type":"ContainerStarted","Data":"6a7425224138f3dd8db65f0ee624d3a21900044da3b0d9dc16e1cbb4902f94a7"} Jul 10 21:21:38.433448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.433397 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" event={"ID":"de2aecbb-9a62-48c4-887b-92b03ac8392b","Type":"ContainerStarted","Data":"b590972903c3415911d6e6d7b72aa419a52b6413f64b8e7a0a8aa702ebbddb6d"} Jul 10 21:21:38.434642 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.434612 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t7db" event={"ID":"a256bf08-cc43-4cae-ade0-910b4772530d","Type":"ContainerStarted","Data":"21d55ed178dbaa442beaf242d8ba8dea2a5a71b3c3edff31577c262161bd3674"} Jul 10 21:21:38.436659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.436610 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" event={"ID":"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d","Type":"ContainerStarted","Data":"15fce1f0094f29f2ba47e3262da751b34da2d6ff10f33519b8f7b20afa3ab6e7"} Jul 10 21:21:38.437868 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.437842 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" event={"ID":"9781f26e-f121-4bdf-a5df-6507501cf548","Type":"ContainerStarted","Data":"dc495075ba5efc5d4a7bd5367d602c5b2d81bc21f1550d84855361aaf8334155"} Jul 10 21:21:38.739295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.739201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:38.739467 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.739355 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:38.739467 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.739414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:38.739594 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.739468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:38.739646 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.739634 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.739701 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.739682223 +0000 UTC m=+39.090205782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740226 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.740207157 +0000 UTC m=+39.090730718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740239 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740241 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740256 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740309 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.740292443 +0000 UTC m=+39.090816018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:38.740355 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.740339 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.740326686 +0000 UTC m=+39.090850242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:38.840511 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.840470 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:38.840743 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.840725 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:38.840810 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.840797 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.840777522 +0000 UTC m=+39.191301081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.941165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:38.941238 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.941385 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.941450 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.941430637 +0000 UTC m=+39.291954208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.941846 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:38.941955 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:38.941895 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:40.941880646 +0000 UTC m=+39.292404206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:40.762793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.762725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.762816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.762867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.762916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763185 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.763166026 +0000 UTC m=+43.113689581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763285 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763299 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763336 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.763324577 +0000 UTC m=+43.113848136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763406 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:40.763439 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763429 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.763421489 +0000 UTC m=+43.113945057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:40.763908 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763473 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:40.763908 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.763514 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.763507267 +0000 UTC m=+43.114030823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:40.863967 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.863934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:40.864157 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.864056 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:40.864157 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.864125 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.864110959 +0000 UTC m=+43.214634515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:40.964805 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.964764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:40.964969 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:40.964829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:40.964969 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.964956 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:40.965043 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.965004 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:40.965101 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.965035 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.965014805 +0000 UTC m=+43.315538366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:40.965101 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:40.965081 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:44.965051689 +0000 UTC m=+43.315575251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:44.803333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:44.803283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:44.803378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:44.803431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803459 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:44.803474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803485 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803540 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803561 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:21:52.80354044 +0000 UTC m=+51.154064016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803607 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:52.803589081 +0000 UTC m=+51.154112656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803612 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803629 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:52.803619279 +0000 UTC m=+51.154142835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:44.803799 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.803673 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:52.803657566 +0000 UTC m=+51.154181139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:44.904329 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:44.904267 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:44.904479 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.904437 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:44.904555 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:44.904514 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:21:52.904493715 +0000 UTC m=+51.255017297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:45.005777 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:45.005738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:45.005951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:45.005809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:45.005951 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:45.005907 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:45.005951 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:45.005930 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:45.006103 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:45.005976 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:53.005960076 +0000 UTC m=+51.356483632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:45.006103 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:45.005993 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:21:53.005984998 +0000 UTC m=+51.356508553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:47.476241 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.476192 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" event={"ID":"de2aecbb-9a62-48c4-887b-92b03ac8392b","Type":"ContainerStarted","Data":"47797b071df080fc48d6dd2f7558dd6524ec3cc3c37f217caa8d72caf19d26ec"} Jul 10 21:21:47.479101 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.479051 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="8d62de642df25391691a43294dcaeaada0b8297ed4956876515d135f0993048f" exitCode=0 Jul 10 21:21:47.479480 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.479367 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"8d62de642df25391691a43294dcaeaada0b8297ed4956876515d135f0993048f"} Jul 10 21:21:47.481511 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.481485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t7db" event={"ID":"a256bf08-cc43-4cae-ade0-910b4772530d","Type":"ContainerStarted","Data":"cbfe43cafee8c8c55d8a2ff38b3f3c34cd6dc12f2e24f5cebfe9e65473b56e9c"} Jul 10 21:21:47.485550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.483341 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" event={"ID":"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d","Type":"ContainerStarted","Data":"ad10de6c7874a39b91fcf9df71570a00c66921f415e265dd88f4e0b1dda409ff"} Jul 10 21:21:47.486260 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.486239 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" event={"ID":"9781f26e-f121-4bdf-a5df-6507501cf548","Type":"ContainerStarted","Data":"354be0eff465a3d2fe6f47142efce120b238ffc67a2d4871ba8ccb0f193b4b73"} Jul 10 21:21:47.488515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.488224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" event={"ID":"ebbffab9-4f2b-4ad9-8706-ed656e2ee5cb","Type":"ContainerStarted","Data":"45010c0406feb25461e32d65268f26ce1739060d4bc62d50d4af2cb117e00239"} Jul 10 21:21:47.489986 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.489926 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" event={"ID":"92294bbd-9727-4430-9c3b-875d7c67e510","Type":"ContainerStarted","Data":"c37b5538b48ebb3ed98f8cada15a61c3d2c647e8735f476ceb4760ad8e29cc38"} Jul 10 21:21:47.491908 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.491887 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/0.log" Jul 10 21:21:47.492018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.491929 2561 generic.go:356] "Generic (PLEG): container finished" podID="f47a0d11-6bb6-4368-812b-c41aa6b2ccbb" containerID="f00c87ba2c841655ebbe3082d22c3e152c9aa1caa57612bd6868f2672a159f44" exitCode=255 Jul 10 21:21:47.492018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.491975 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d642m" event={"ID":"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb","Type":"ContainerDied","Data":"f00c87ba2c841655ebbe3082d22c3e152c9aa1caa57612bd6868f2672a159f44"} Jul 10 21:21:47.492258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.492242 2561 scope.go:117] "RemoveContainer" containerID="f00c87ba2c841655ebbe3082d22c3e152c9aa1caa57612bd6868f2672a159f44" Jul 10 21:21:47.498098 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.497912 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" podStartSLOduration=34.526563873 podStartE2EDuration="43.497893962s" totalImagesPullingTime="8.971330089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.497170441 +0000 UTC m=+45.847694012" watchObservedRunningTime="2026-07-10 21:21:47.497893962 +0000 UTC m=+45.848417520" Jul 10 21:21:47.579466 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.579394 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5t7db" podStartSLOduration=2.686240982 podStartE2EDuration="11.579371025s" totalImagesPullingTime="8.893130043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.546814941 +0000 UTC m=+45.897338524" watchObservedRunningTime="2026-07-10 21:21:47.579371025 +0000 UTC m=+45.929894604" Jul 10 21:21:47.619873 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.618977 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-hj44q" podStartSLOduration=34.697064703 podStartE2EDuration="43.618957066s" totalImagesPullingTime="8.921892363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.598171945 +0000 UTC m=+45.948695551" watchObservedRunningTime="2026-07-10 21:21:47.618957066 +0000 UTC m=+45.969480647" Jul 10 21:21:47.619873 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.619449 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ljr87" podStartSLOduration=31.678439052999998 podStartE2EDuration="40.619432157s" totalImagesPullingTime="8.940993104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.619284938 +0000 UTC m=+45.969808516" watchObservedRunningTime="2026-07-10 21:21:47.619432157 +0000 UTC m=+45.969955736" Jul 10 21:21:47.689954 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.689890 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" podStartSLOduration=34.640060609 podStartE2EDuration="43.689866418s" totalImagesPullingTime="9.049805809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.664268969 +0000 UTC m=+46.014792547" watchObservedRunningTime="2026-07-10 21:21:47.689866418 +0000 UTC m=+46.040389997" Jul 10 21:21:47.691241 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:47.691196 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" podStartSLOduration=34.634584571 podStartE2EDuration="43.691179187s" totalImagesPullingTime="9.056594616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:47.689158897 +0000 UTC m=+46.039682477" watchObservedRunningTime="2026-07-10 21:21:47.691179187 +0000 UTC m=+46.041702765" Jul 10 21:21:48.497288 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.497253 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:21:48.497703 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.497684 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/0.log" Jul 10 21:21:48.497758 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.497724 2561 generic.go:356] "Generic (PLEG): container finished" podID="f47a0d11-6bb6-4368-812b-c41aa6b2ccbb" containerID="9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8" exitCode=255 Jul 10 21:21:48.497856 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.497821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d642m" event={"ID":"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb","Type":"ContainerDied","Data":"9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8"} Jul 10 21:21:48.497994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.497887 2561 scope.go:117] "RemoveContainer" containerID="f00c87ba2c841655ebbe3082d22c3e152c9aa1caa57612bd6868f2672a159f44" Jul 10 21:21:48.498191 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.498175 2561 scope.go:117] "RemoveContainer" containerID="9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8" Jul 10 21:21:48.498427 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:48.498404 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d642m_openshift-console-operator(f47a0d11-6bb6-4368-812b-c41aa6b2ccbb)\"" pod="openshift-console-operator/console-operator-7866497494-d642m" podUID="f47a0d11-6bb6-4368-812b-c41aa6b2ccbb" Jul 10 21:21:48.501702 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.501677 2561 generic.go:356] "Generic (PLEG): container finished" podID="5014c522-e157-46e0-9fab-5df447a1d745" containerID="4c1169734f76804b0bc31fde80af03414cc9ddc8ba6caeb4a2794dddb7924631" exitCode=0 Jul 10 21:21:48.501839 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:48.501814 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerDied","Data":"4c1169734f76804b0bc31fde80af03414cc9ddc8ba6caeb4a2794dddb7924631"} Jul 10 21:21:49.190410 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.190377 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg"] Jul 10 21:21:49.192972 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.192955 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" Jul 10 21:21:49.195566 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.195534 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 21:21:49.196719 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.196700 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 21:21:49.196820 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.196761 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ldwvv\"" Jul 10 21:21:49.208133 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.208108 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg"] Jul 10 21:21:49.245807 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.245765 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ddz\" (UniqueName: \"kubernetes.io/projected/2e1cf7a2-c69d-46fb-bd7a-3d27aed69252-kube-api-access-m7ddz\") pod \"migrator-7f59b54f6c-4hrsg\" (UID: \"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" Jul 10 21:21:49.347142 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.347092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ddz\" (UniqueName: \"kubernetes.io/projected/2e1cf7a2-c69d-46fb-bd7a-3d27aed69252-kube-api-access-m7ddz\") pod \"migrator-7f59b54f6c-4hrsg\" (UID: \"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" Jul 10 21:21:49.357649 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.357610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ddz\" (UniqueName: \"kubernetes.io/projected/2e1cf7a2-c69d-46fb-bd7a-3d27aed69252-kube-api-access-m7ddz\") pod \"migrator-7f59b54f6c-4hrsg\" (UID: \"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" Jul 10 21:21:49.503647 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.503558 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" Jul 10 21:21:49.512311 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.512268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zczs" event={"ID":"5014c522-e157-46e0-9fab-5df447a1d745","Type":"ContainerStarted","Data":"b11dc7698e8b8f7bf485beee53f6a840c83d0f6ac7d8e39d90b095dd182dcec2"} Jul 10 21:21:49.514222 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.514200 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:21:49.514558 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.514537 2561 scope.go:117] "RemoveContainer" containerID="9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8" Jul 10 21:21:49.514749 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:49.514728 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d642m_openshift-console-operator(f47a0d11-6bb6-4368-812b-c41aa6b2ccbb)\"" pod="openshift-console-operator/console-operator-7866497494-d642m" podUID="f47a0d11-6bb6-4368-812b-c41aa6b2ccbb" Jul 10 21:21:49.547708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.547671 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rnhh5"] Jul 10 21:21:49.551033 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.551001 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.554434 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.554407 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 21:21:49.554586 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.554526 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f6m5r\"" Jul 10 21:21:49.554647 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.554407 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 21:21:49.555194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.555151 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2zczs" podStartSLOduration=10.133427816 podStartE2EDuration="47.555135866s" totalImagesPullingTime="37.42170805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:49.554529085 +0000 UTC m=+47.905052690" watchObservedRunningTime="2026-07-10 21:21:49.555135866 +0000 UTC m=+47.905659421" Jul 10 21:21:49.564769 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.564743 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rnhh5"] Jul 10 21:21:49.649274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.649240 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1ca98a1-b383-43b2-b060-b33658e0b5aa-crio-socket\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.649525 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.649306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.649525 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.649372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.649525 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.649403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b78p\" (UniqueName: \"kubernetes.io/projected/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-api-access-4b78p\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.649682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.649546 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1ca98a1-b383-43b2-b060-b33658e0b5aa-data-volume\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.676298 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.676250 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg"] Jul 10 21:21:49.679577 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:21:49.679537 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1cf7a2_c69d_46fb_bd7a_3d27aed69252.slice/crio-322cc2144e51e52eb56a670b03c022707e4ab24bcf584596a6d9913939e58853 WatchSource:0}: Error finding container 322cc2144e51e52eb56a670b03c022707e4ab24bcf584596a6d9913939e58853: Status 404 returned error can't find the container with id 322cc2144e51e52eb56a670b03c022707e4ab24bcf584596a6d9913939e58853 Jul 10 21:21:49.750803 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.750766 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.750976 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.750828 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.750976 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:49.750911 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 21:21:49.750976 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:49.750919 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 21:21:49.750976 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.750960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b78p\" (UniqueName: \"kubernetes.io/projected/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-api-access-4b78p\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.751233 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:49.750984 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:50.250964275 +0000 UTC m=+48.601487846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : configmap "kube-rbac-proxy" not found Jul 10 21:21:49.751233 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:49.751019 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:50.251005421 +0000 UTC m=+48.601528979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : secret "insights-runtime-extractor-tls" not found Jul 10 21:21:49.751233 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.751140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1ca98a1-b383-43b2-b060-b33658e0b5aa-data-volume\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.751233 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.751183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1ca98a1-b383-43b2-b060-b33658e0b5aa-crio-socket\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.751388 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.751331 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1ca98a1-b383-43b2-b060-b33658e0b5aa-crio-socket\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.751425 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.751394 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1ca98a1-b383-43b2-b060-b33658e0b5aa-data-volume\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:49.760111 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:49.760022 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b78p\" (UniqueName: \"kubernetes.io/projected/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-api-access-4b78p\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:50.255749 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:50.255696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:50.255926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:50.255772 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:50.255926 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:50.255840 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 21:21:50.255926 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:50.255920 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:51.255903578 +0000 UTC m=+49.606427138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : configmap "kube-rbac-proxy" not found Jul 10 21:21:50.256098 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:50.255936 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 21:21:50.256098 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:50.255985 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:51.255972136 +0000 UTC m=+49.606495695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : secret "insights-runtime-extractor-tls" not found Jul 10 21:21:50.519442 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:50.519356 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" event={"ID":"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252","Type":"ContainerStarted","Data":"322cc2144e51e52eb56a670b03c022707e4ab24bcf584596a6d9913939e58853"} Jul 10 21:21:51.267519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.267414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:51.267519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.267482 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:51.267761 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:51.267565 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 21:21:51.267761 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:51.267600 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 21:21:51.267761 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:51.267648 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:53.267628451 +0000 UTC m=+51.618152007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : configmap "kube-rbac-proxy" not found Jul 10 21:21:51.267761 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:51.267667 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:53.267658663 +0000 UTC m=+51.618182227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : secret "insights-runtime-extractor-tls" not found Jul 10 21:21:51.341793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.341760 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-649j8_fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc/dns-node-resolver/0.log" Jul 10 21:21:51.524279 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.524187 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" event={"ID":"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252","Type":"ContainerStarted","Data":"a1b947dd0df03ccf614fe11adf97dfcd617f88bb956f7fdc914674ec44a4eb21"} Jul 10 21:21:51.524279 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.524238 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" event={"ID":"2e1cf7a2-c69d-46fb-bd7a-3d27aed69252","Type":"ContainerStarted","Data":"96306651fb19f471bdd19ff98860b4189ace79f23863d956917d5c59f0a0470c"} Jul 10 21:21:51.544880 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:51.544824 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-4hrsg" podStartSLOduration=1.3588602060000001 podStartE2EDuration="2.544806621s" totalImagesPullingTime="1.185946415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:51.543907017 +0000 UTC m=+49.894430618" watchObservedRunningTime="2026-07-10 21:21:51.544806621 +0000 UTC m=+49.895330198" Jul 10 21:21:52.742051 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.742019 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cq2g5_3229741a-df29-40c4-9abe-18205becc9d0/node-ca/0.log" Jul 10 21:21:52.880246 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.880201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.880266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.880304 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.880343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880378 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880400 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b5c586bd-5d9xg: secret "image-registry-tls" not found Jul 10 21:21:52.880452 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880429 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:52.880746 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880451 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 21:21:52.880746 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880468 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls podName:db491d18-185b-417c-a213-b80ddb40c74b nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.880446288 +0000 UTC m=+67.230969858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls") pod "image-registry-66b5c586bd-5d9xg" (UID: "db491d18-185b-417c-a213-b80ddb40c74b") : secret "image-registry-tls" not found Jul 10 21:21:52.880746 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880488 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.88047915 +0000 UTC m=+67.231002707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : configmap references non-existent config key: service-ca.crt Jul 10 21:21:52.880746 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880508 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs podName:32d29f1c-89cd-42f6-b9b6-e08b28e47266 nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.880494409 +0000 UTC m=+67.231017970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs") pod "router-default-68f44597c6-hzxv6" (UID: "32d29f1c-89cd-42f6-b9b6-e08b28e47266") : secret "router-metrics-certs-default" not found Jul 10 21:21:52.880746 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.880610 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls podName:4fb4b267-19c8-4f61-b649-35dfff7bec04 nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.880584744 +0000 UTC m=+67.231108303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-s7sww" (UID: "4fb4b267-19c8-4f61-b649-35dfff7bec04") : secret "cluster-monitoring-operator-tls" not found Jul 10 21:21:52.981371 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:52.981332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:21:52.981547 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.981474 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 21:21:52.981547 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:52.981531 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls podName:28c084d6-5a77-48db-a48a-6bf88d2d91be nodeName:}" failed. No retries permitted until 2026-07-10 21:22:08.981517723 +0000 UTC m=+67.332041280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls") pod "cluster-samples-operator-65db858894-9nnxd" (UID: "28c084d6-5a77-48db-a48a-6bf88d2d91be") : secret "samples-operator-tls" not found Jul 10 21:21:53.082228 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.082124 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:21:53.082394 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.082284 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 21:21:53.082394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.082301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:21:53.082394 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.082353 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert podName:ac6d2700-7161-4eea-a941-329fa08026ac nodeName:}" failed. No retries permitted until 2026-07-10 21:22:09.08233748 +0000 UTC m=+67.432861035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-m5bj4" (UID: "ac6d2700-7161-4eea-a941-329fa08026ac") : secret "networking-console-plugin-cert" not found Jul 10 21:21:53.082394 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.082392 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 21:21:53.082523 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.082433 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls podName:acb3f997-86fd-499f-9b1b-f9b657fbfdac nodeName:}" failed. No retries permitted until 2026-07-10 21:22:09.082422437 +0000 UTC m=+67.432945993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls") pod "dns-default-9x8k5" (UID: "acb3f997-86fd-499f-9b1b-f9b657fbfdac") : secret "dns-default-metrics-tls" not found Jul 10 21:21:53.142481 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.142447 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5t7db_a256bf08-cc43-4cae-ade0-910b4772530d/serve-healthcheck-canary/0.log" Jul 10 21:21:53.284010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.283965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:53.284198 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.284132 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 21:21:53.284198 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.284187 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 21:21:53.284272 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.284193 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:57.284178265 +0000 UTC m=+55.634701821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : secret "insights-runtime-extractor-tls" not found Jul 10 21:21:53.284272 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:53.284254 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:21:57.284241554 +0000 UTC m=+55.634765110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : configmap "kube-rbac-proxy" not found Jul 10 21:21:53.284272 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.284139 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:53.342967 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.342869 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-4hrsg_2e1cf7a2-c69d-46fb-bd7a-3d27aed69252/migrator/0.log" Jul 10 21:21:53.541032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.541000 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-4hrsg_2e1cf7a2-c69d-46fb-bd7a-3d27aed69252/graceful-termination/0.log" Jul 10 21:21:53.743293 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:53.743251 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-5hgv9_9781f26e-f121-4bdf-a5df-6507501cf548/kube-storage-version-migrator-operator/0.log" Jul 10 21:21:57.319354 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.319314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:57.319942 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.319379 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:57.319942 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:57.319545 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 21:21:57.319942 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:57.319615 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls podName:b1ca98a1-b383-43b2-b060-b33658e0b5aa nodeName:}" failed. No retries permitted until 2026-07-10 21:22:05.319595997 +0000 UTC m=+63.670119577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnhh5" (UID: "b1ca98a1-b383-43b2-b060-b33658e0b5aa") : secret "insights-runtime-extractor-tls" not found Jul 10 21:21:57.320136 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.319969 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1ca98a1-b383-43b2-b060-b33658e0b5aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:21:57.416949 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.416899 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:57.417148 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.416983 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:21:57.417417 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:21:57.417401 2561 scope.go:117] "RemoveContainer" containerID="9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8" Jul 10 21:21:57.417604 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:21:57.417583 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-d642m_openshift-console-operator(f47a0d11-6bb6-4368-812b-c41aa6b2ccbb)\"" pod="openshift-console-operator/console-operator-7866497494-d642m" podUID="f47a0d11-6bb6-4368-812b-c41aa6b2ccbb" Jul 10 21:22:04.426726 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:04.426696 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t9277" Jul 10 21:22:05.399608 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:05.399565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:22:05.402005 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:05.401983 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1ca98a1-b383-43b2-b060-b33658e0b5aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnhh5\" (UID: \"b1ca98a1-b383-43b2-b060-b33658e0b5aa\") " pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:22:05.464129 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:05.464095 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f6m5r\"" Jul 10 21:22:05.472138 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:05.472112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rnhh5" Jul 10 21:22:05.656089 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:05.655841 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rnhh5"] Jul 10 21:22:05.658694 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:05.658667 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ca98a1_b383_43b2_b060_b33658e0b5aa.slice/crio-e4da6ea7fce80760efd4dda6938e91f16acc3545172b6daced1c138f710e34b3 WatchSource:0}: Error finding container e4da6ea7fce80760efd4dda6938e91f16acc3545172b6daced1c138f710e34b3: Status 404 returned error can't find the container with id e4da6ea7fce80760efd4dda6938e91f16acc3545172b6daced1c138f710e34b3 Jul 10 21:22:06.572639 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:06.572594 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnhh5" event={"ID":"b1ca98a1-b383-43b2-b060-b33658e0b5aa","Type":"ContainerStarted","Data":"e4da6ea7fce80760efd4dda6938e91f16acc3545172b6daced1c138f710e34b3"} Jul 10 21:22:07.922272 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.922236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:22:07.922732 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.922314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:22:07.927406 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.927378 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 21:22:07.928804 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.928782 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 21:22:07.935553 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.935528 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d66904b-9304-4ab3-89e9-042ad744cbb3-metrics-certs\") pod \"network-metrics-daemon-vfl8r\" (UID: \"7d66904b-9304-4ab3-89e9-042ad744cbb3\") " pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:22:07.935683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:07.935569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64784df-28d5-4bb5-b363-9169dd0308af-original-pull-secret\") pod \"global-pull-secret-syncer-989qr\" (UID: \"b64784df-28d5-4bb5-b363-9169dd0308af\") " pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:22:08.023670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.023625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:22:08.026801 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.026743 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbb2z\" (UniqueName: \"kubernetes.io/projected/c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a-kube-api-access-qbb2z\") pod \"network-check-target-8gqlw\" (UID: \"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a\") " pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:22:08.192777 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.192686 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j44v2\"" Jul 10 21:22:08.198351 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.198323 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nklrg\"" Jul 10 21:22:08.200441 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.200401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfl8r" Jul 10 21:22:08.200594 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.200511 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-989qr" Jul 10 21:22:08.206890 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.206860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:22:08.932672 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.932619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:08.933140 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.932751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:22:08.933140 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.932808 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:22:08.933140 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.932857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:08.933431 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.933402 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d29f1c-89cd-42f6-b9b6-e08b28e47266-service-ca-bundle\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:08.936216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.936178 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d29f1c-89cd-42f6-b9b6-e08b28e47266-metrics-certs\") pod \"router-default-68f44597c6-hzxv6\" (UID: \"32d29f1c-89cd-42f6-b9b6-e08b28e47266\") " pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:08.936216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.936186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"image-registry-66b5c586bd-5d9xg\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:22:08.936400 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:08.936178 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fb4b267-19c8-4f61-b649-35dfff7bec04-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-s7sww\" (UID: \"4fb4b267-19c8-4f61-b649-35dfff7bec04\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:22:09.034220 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.034187 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:22:09.036988 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.036954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28c084d6-5a77-48db-a48a-6bf88d2d91be-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9nnxd\" (UID: \"28c084d6-5a77-48db-a48a-6bf88d2d91be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:22:09.107885 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.107852 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lkx9k\"" Jul 10 21:22:09.115413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.115379 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:22:09.125927 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.125893 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4nbjf\"" Jul 10 21:22:09.132330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.132286 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" Jul 10 21:22:09.135588 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.135544 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:22:09.135691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.135604 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:22:09.138201 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.138176 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q5rzv\"" Jul 10 21:22:09.138346 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.138327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acb3f997-86fd-499f-9b1b-f9b657fbfdac-metrics-tls\") pod \"dns-default-9x8k5\" (UID: \"acb3f997-86fd-499f-9b1b-f9b657fbfdac\") " pod="openshift-dns/dns-default-9x8k5" Jul 10 21:22:09.138641 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.138617 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac6d2700-7161-4eea-a941-329fa08026ac-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-m5bj4\" (UID: \"ac6d2700-7161-4eea-a941-329fa08026ac\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:22:09.145987 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.145923 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:09.196393 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.196304 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dtwtt\"" Jul 10 21:22:09.203984 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.203934 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" Jul 10 21:22:09.301932 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.301905 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ns8ld\"" Jul 10 21:22:09.310219 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.310186 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" Jul 10 21:22:09.356990 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.356955 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hw29l\"" Jul 10 21:22:09.365168 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.365136 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x8k5" Jul 10 21:22:09.876569 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.876454 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gqlw"] Jul 10 21:22:09.886301 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:09.886175 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b3c8f_f8a2_4f5e_a8d0_6aafad92250a.slice/crio-3b4577f1197b1e05de83359a3ad3ff7fee051439220358fbfd7932346af4b2bb WatchSource:0}: Error finding container 3b4577f1197b1e05de83359a3ad3ff7fee051439220358fbfd7932346af4b2bb: Status 404 returned error can't find the container with id 3b4577f1197b1e05de83359a3ad3ff7fee051439220358fbfd7932346af4b2bb Jul 10 21:22:09.977176 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.976770 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfl8r"] Jul 10 21:22:09.983336 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.981745 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f44597c6-hzxv6"] Jul 10 21:22:09.984279 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:09.983739 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x8k5"] Jul 10 21:22:10.006879 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.006790 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-989qr"] Jul 10 21:22:10.015854 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:10.014361 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64784df_28d5_4bb5_b363_9169dd0308af.slice/crio-88738db12ec41d79a2dffd1895d67d3f28e33da1a88976d000e7a069569d885a WatchSource:0}: Error finding container 88738db12ec41d79a2dffd1895d67d3f28e33da1a88976d000e7a069569d885a: Status 404 returned error can't find the container with id 88738db12ec41d79a2dffd1895d67d3f28e33da1a88976d000e7a069569d885a Jul 10 21:22:10.036041 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.035840 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-m5bj4"] Jul 10 21:22:10.038489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.038462 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww"] Jul 10 21:22:10.043410 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:10.043379 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb4b267_19c8_4f61_b649_35dfff7bec04.slice/crio-52d132d7d55e6410425e5855c7339afbb323183ab650a291898ebbacd3e25229 WatchSource:0}: Error finding container 52d132d7d55e6410425e5855c7339afbb323183ab650a291898ebbacd3e25229: Status 404 returned error can't find the container with id 52d132d7d55e6410425e5855c7339afbb323183ab650a291898ebbacd3e25229 Jul 10 21:22:10.043575 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:10.043551 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6d2700_7161_4eea_a941_329fa08026ac.slice/crio-c15992f7fc04d20691c799346fa0328549ec7935e35a253c2b312d8c7365149a WatchSource:0}: Error finding container c15992f7fc04d20691c799346fa0328549ec7935e35a253c2b312d8c7365149a: Status 404 returned error can't find the container with id c15992f7fc04d20691c799346fa0328549ec7935e35a253c2b312d8c7365149a Jul 10 21:22:10.075443 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.075411 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:22:10.078417 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:10.078388 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb491d18_185b_417c_a213_b80ddb40c74b.slice/crio-1b92c6a9bdf352ae2060c94efd85624083ab4a2ad64de51384d4bb541ad51b68 WatchSource:0}: Error finding container 1b92c6a9bdf352ae2060c94efd85624083ab4a2ad64de51384d4bb541ad51b68: Status 404 returned error can't find the container with id 1b92c6a9bdf352ae2060c94efd85624083ab4a2ad64de51384d4bb541ad51b68 Jul 10 21:22:10.227359 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.227324 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd"] Jul 10 21:22:10.588939 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.588859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" event={"ID":"28c084d6-5a77-48db-a48a-6bf88d2d91be","Type":"ContainerStarted","Data":"fe6d458ab6c62e44647a003ac7bd2b7214261e87c4e9c579a3dbdc01b583a13a"} Jul 10 21:22:10.590208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.590183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfl8r" event={"ID":"7d66904b-9304-4ab3-89e9-042ad744cbb3","Type":"ContainerStarted","Data":"60f1c8d7c95885a742c163126eb6b831313fb2f0ed12c21c9d273f91cd19cba0"} Jul 10 21:22:10.592787 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.592756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnhh5" event={"ID":"b1ca98a1-b383-43b2-b060-b33658e0b5aa","Type":"ContainerStarted","Data":"db984f9e3359f7c81be9292d82dce35e9e0107e91426d837d507e7d0c4e160b7"} Jul 10 21:22:10.592894 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.592815 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnhh5" event={"ID":"b1ca98a1-b383-43b2-b060-b33658e0b5aa","Type":"ContainerStarted","Data":"5aebfd09cbbaf1281f50e4dcc456273e887b7b87d9ca28fe4b8f9dcc8bf4ae11"} Jul 10 21:22:10.594354 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.594329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x8k5" event={"ID":"acb3f997-86fd-499f-9b1b-f9b657fbfdac","Type":"ContainerStarted","Data":"b8f6a1b8e2e79f258cfb05cc10839733fd4c01f8b4d8eca0f7d765029fdbe4a5"} Jul 10 21:22:10.595853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.595828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-989qr" event={"ID":"b64784df-28d5-4bb5-b363-9169dd0308af","Type":"ContainerStarted","Data":"88738db12ec41d79a2dffd1895d67d3f28e33da1a88976d000e7a069569d885a"} Jul 10 21:22:10.597188 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.597143 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" event={"ID":"ac6d2700-7161-4eea-a941-329fa08026ac","Type":"ContainerStarted","Data":"c15992f7fc04d20691c799346fa0328549ec7935e35a253c2b312d8c7365149a"} Jul 10 21:22:10.598772 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.598722 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" event={"ID":"4fb4b267-19c8-4f61-b649-35dfff7bec04","Type":"ContainerStarted","Data":"52d132d7d55e6410425e5855c7339afbb323183ab650a291898ebbacd3e25229"} Jul 10 21:22:10.600776 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.600707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" event={"ID":"db491d18-185b-417c-a213-b80ddb40c74b","Type":"ContainerStarted","Data":"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63"} Jul 10 21:22:10.600776 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.600746 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" event={"ID":"db491d18-185b-417c-a213-b80ddb40c74b","Type":"ContainerStarted","Data":"1b92c6a9bdf352ae2060c94efd85624083ab4a2ad64de51384d4bb541ad51b68"} Jul 10 21:22:10.601139 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.601100 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:22:10.603714 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.603458 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gqlw" event={"ID":"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a","Type":"ContainerStarted","Data":"90299fad3f28661b979f4c0fed36cdc20fe082b453037e23fe898bebf34f6d48"} Jul 10 21:22:10.603714 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.603508 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:22:10.603714 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.603531 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gqlw" event={"ID":"c65b3c8f-f8a2-4f5e-a8d0-6aafad92250a","Type":"ContainerStarted","Data":"3b4577f1197b1e05de83359a3ad3ff7fee051439220358fbfd7932346af4b2bb"} Jul 10 21:22:10.606192 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.606168 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f44597c6-hzxv6" event={"ID":"32d29f1c-89cd-42f6-b9b6-e08b28e47266","Type":"ContainerStarted","Data":"ff62d9441cc0522bce1c5df51d67ed314d6f479e431f4ccff1ee1b3d9b3d2c47"} Jul 10 21:22:10.606307 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.606211 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f44597c6-hzxv6" event={"ID":"32d29f1c-89cd-42f6-b9b6-e08b28e47266","Type":"ContainerStarted","Data":"6ce9a6f5d2988a241acacce5de010600e48030204b5c194eb616af0455dfca5b"} Jul 10 21:22:10.625004 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.624933 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" podStartSLOduration=67.62490988 podStartE2EDuration="1m7.62490988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:10.623488477 +0000 UTC m=+68.974012077" watchObservedRunningTime="2026-07-10 21:22:10.62490988 +0000 UTC m=+68.975433462" Jul 10 21:22:10.652842 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.652785 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68f44597c6-hzxv6" podStartSLOduration=66.652766172 podStartE2EDuration="1m6.652766172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:10.651481541 +0000 UTC m=+69.002005120" watchObservedRunningTime="2026-07-10 21:22:10.652766172 +0000 UTC m=+69.003289788" Jul 10 21:22:10.675792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:10.675232 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gqlw" podStartSLOduration=68.675212075 podStartE2EDuration="1m8.675212075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:10.673452463 +0000 UTC m=+69.023976061" watchObservedRunningTime="2026-07-10 21:22:10.675212075 +0000 UTC m=+69.025735655" Jul 10 21:22:11.146565 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:11.146488 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:11.150377 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:11.150152 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:11.182824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:11.180720 2561 scope.go:117] "RemoveContainer" containerID="9722822eeb576eccfe16781579b036acc1cd26300a5a97c123cc92d0e531a3e8" Jul 10 21:22:11.613446 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:11.613210 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:11.614994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:11.614742 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68f44597c6-hzxv6" Jul 10 21:22:12.597797 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.597761 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q"] Jul 10 21:22:12.601184 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.600497 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.604249 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.603545 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 21:22:12.604929 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.604554 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 21:22:12.604929 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.604648 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 21:22:12.604929 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.604753 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 21:22:12.620285 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.620254 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q"] Jul 10 21:22:12.670611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.670576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d11f6152-0ba8-4cb9-a306-6607c569bce7-klusterlet-config\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.670793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.670649 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckcv\" (UniqueName: \"kubernetes.io/projected/d11f6152-0ba8-4cb9-a306-6607c569bce7-kube-api-access-gckcv\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.670793 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.670719 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d11f6152-0ba8-4cb9-a306-6607c569bce7-tmp\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.771981 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.771939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckcv\" (UniqueName: \"kubernetes.io/projected/d11f6152-0ba8-4cb9-a306-6607c569bce7-kube-api-access-gckcv\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.772203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.772010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d11f6152-0ba8-4cb9-a306-6607c569bce7-tmp\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.772203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.772126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d11f6152-0ba8-4cb9-a306-6607c569bce7-klusterlet-config\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.772998 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.772968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d11f6152-0ba8-4cb9-a306-6607c569bce7-tmp\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.775368 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.775347 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d11f6152-0ba8-4cb9-a306-6607c569bce7-klusterlet-config\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.824374 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.824288 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckcv\" (UniqueName: \"kubernetes.io/projected/d11f6152-0ba8-4cb9-a306-6607c569bce7-kube-api-access-gckcv\") pod \"klusterlet-addon-workmgr-677c85c8cf-47q4q\" (UID: \"d11f6152-0ba8-4cb9-a306-6607c569bce7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:12.920306 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:12.920259 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:16.552862 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.552807 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q"] Jul 10 21:22:16.645114 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.645023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" event={"ID":"28c084d6-5a77-48db-a48a-6bf88d2d91be","Type":"ContainerStarted","Data":"f975b665078a8b1d7f79cf5a3eae5f79b3b2725bed00cfb2b1f7ca860179a287"} Jul 10 21:22:16.645287 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.645154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" event={"ID":"28c084d6-5a77-48db-a48a-6bf88d2d91be","Type":"ContainerStarted","Data":"0053ab74118b2c2ee33c5e91584333dece316cf43c745a81dcb1bce7653fadb2"} Jul 10 21:22:16.660425 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.660294 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfl8r" event={"ID":"7d66904b-9304-4ab3-89e9-042ad744cbb3","Type":"ContainerStarted","Data":"f29f91aea30c4c0d86f8d08b2a87875501bb3ff6de11182cacceb12ff77d62b7"} Jul 10 21:22:16.683322 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.679475 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9nnxd" podStartSLOduration=66.606812241 podStartE2EDuration="1m12.679451467s" totalImagesPullingTime="6.072639226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.678104301 +0000 UTC m=+75.028627910" watchObservedRunningTime="2026-07-10 21:22:16.679451467 +0000 UTC m=+75.029975046" Jul 10 21:22:16.684466 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.684175 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnhh5" event={"ID":"b1ca98a1-b383-43b2-b060-b33658e0b5aa","Type":"ContainerStarted","Data":"462376c84eeb056e82447bd49f4e74de1b2d48e4a91d23c03e6b7d839c1b9625"} Jul 10 21:22:16.691746 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.691650 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x8k5" event={"ID":"acb3f997-86fd-499f-9b1b-f9b657fbfdac","Type":"ContainerStarted","Data":"8ec1e224e8ebfb625af0cce25458681571cf74faf2b17631eea48e90c230d2c8"} Jul 10 21:22:16.694233 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.693676 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-989qr" event={"ID":"b64784df-28d5-4bb5-b363-9169dd0308af","Type":"ContainerStarted","Data":"5dcda8187d0db6797b35a0155dc94c276bc296f1942c688aa7eebde969c14a7e"} Jul 10 21:22:16.697632 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.697179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:22:16.697632 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.697268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-d642m" event={"ID":"f47a0d11-6bb6-4368-812b-c41aa6b2ccbb","Type":"ContainerStarted","Data":"08b649ebe3b5e04403500ce6eb1eda57e265323dcad4373af3940c8a173ab4d8"} Jul 10 21:22:16.698561 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.698498 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:22:16.702397 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.701872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" event={"ID":"ac6d2700-7161-4eea-a941-329fa08026ac","Type":"ContainerStarted","Data":"929cea21790c1f53bd289b0513412946d28140b6615e9dc9a4b7cbdf41b1f6dd"} Jul 10 21:22:16.704657 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.704587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" event={"ID":"4fb4b267-19c8-4f61-b649-35dfff7bec04","Type":"ContainerStarted","Data":"eeba34e189ad9c468cdea8d246b172d0052c0a875128becc17ce90c0435cad50"} Jul 10 21:22:16.709206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.709176 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" event={"ID":"d11f6152-0ba8-4cb9-a306-6607c569bce7","Type":"ContainerStarted","Data":"17ee20743b77db84f639a170fd01bc5b5792c5c13adf0080911139cd58f32859"} Jul 10 21:22:16.709935 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.709880 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rnhh5" podStartSLOduration=17.229017115 podStartE2EDuration="27.709860097s" totalImagesPullingTime="10.480842982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.707650093 +0000 UTC m=+75.058173694" watchObservedRunningTime="2026-07-10 21:22:16.709860097 +0000 UTC m=+75.060383678" Jul 10 21:22:16.730117 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.729608 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-m5bj4" podStartSLOduration=34.458487776 podStartE2EDuration="40.729588739s" totalImagesPullingTime="6.271100963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.727092606 +0000 UTC m=+75.077616195" watchObservedRunningTime="2026-07-10 21:22:16.729588739 +0000 UTC m=+75.080112319" Jul 10 21:22:16.753437 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.751963 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-s7sww" podStartSLOduration=66.44389588 podStartE2EDuration="1m12.751940968s" totalImagesPullingTime="6.308045088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.750976029 +0000 UTC m=+75.101499609" watchObservedRunningTime="2026-07-10 21:22:16.751940968 +0000 UTC m=+75.102464546" Jul 10 21:22:16.804916 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.804605 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-d642m" podStartSLOduration=63.777554232 podStartE2EDuration="1m12.804564192s" totalImagesPullingTime="9.02700996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.776590128 +0000 UTC m=+75.127113724" watchObservedRunningTime="2026-07-10 21:22:16.804564192 +0000 UTC m=+75.155087772" Jul 10 21:22:16.810098 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:16.809632 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-989qr" podStartSLOduration=67.441899865 podStartE2EDuration="1m13.80960981s" totalImagesPullingTime="6.367709945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:16.796846263 +0000 UTC m=+75.147369841" watchObservedRunningTime="2026-07-10 21:22:16.80960981 +0000 UTC m=+75.160133388" Jul 10 21:22:17.267847 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.267806 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-d642m" Jul 10 21:22:17.716936 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.716665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfl8r" event={"ID":"7d66904b-9304-4ab3-89e9-042ad744cbb3","Type":"ContainerStarted","Data":"876bf68ac183fbae3f0904aa6156c1a5216cce629c6a996f4cacbe092df30bf1"} Jul 10 21:22:17.719323 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.719291 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x8k5" event={"ID":"acb3f997-86fd-499f-9b1b-f9b657fbfdac","Type":"ContainerStarted","Data":"26912ea5d4611141ed3995679c6fa5c985f5516d81e50bb9821c2732deb6f297"} Jul 10 21:22:17.720865 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.720125 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9x8k5" Jul 10 21:22:17.736212 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.736133 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vfl8r" podStartSLOduration=69.808317725 podStartE2EDuration="1m14.736111031s" totalImagesPullingTime="4.927793306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:17.734715685 +0000 UTC m=+76.085239283" watchObservedRunningTime="2026-07-10 21:22:17.736111031 +0000 UTC m=+76.086634611" Jul 10 21:22:17.758366 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:17.758299 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9x8k5" podStartSLOduration=35.397644512 podStartE2EDuration="41.758273985s" totalImagesPullingTime="6.360629473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:17.75700849 +0000 UTC m=+76.107532096" watchObservedRunningTime="2026-07-10 21:22:17.758273985 +0000 UTC m=+76.108797566" Jul 10 21:22:20.734506 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:20.734462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" event={"ID":"d11f6152-0ba8-4cb9-a306-6607c569bce7","Type":"ContainerStarted","Data":"a3d9fae188eef99e1dc2661c67bdf29f0bbaff598e50aaf48854039c48b4c06b"} Jul 10 21:22:20.734934 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:20.734663 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:20.736496 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:20.736470 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" Jul 10 21:22:20.753277 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:20.753229 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677c85c8cf-47q4q" podStartSLOduration=4.758934272 podStartE2EDuration="8.753211623s" totalImagesPullingTime="3.994277351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:20.752905737 +0000 UTC m=+79.103429315" watchObservedRunningTime="2026-07-10 21:22:20.753211623 +0000 UTC m=+79.103735278" Jul 10 21:22:28.691760 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.691636 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x9hh7"] Jul 10 21:22:28.694807 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.694786 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.699150 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.699124 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 21:22:28.699278 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.699171 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 21:22:28.700171 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.700142 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 21:22:28.701010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.700989 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5pwrj\"" Jul 10 21:22:28.701144 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.701128 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 21:22:28.731776 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.731744 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9x8k5" Jul 10 21:22:28.805279 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805247 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-wtmp\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805279 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805311 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805340 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-metrics-client-ca\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805383 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-sys\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805405 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-root\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805478 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvkm\" (UniqueName: \"kubernetes.io/projected/b7caf5ef-960a-41c9-9b50-872641002718-kube-api-access-8zvkm\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805518 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-textfile\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.805728 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.805602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-tls\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-sys\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-root\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-root\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906498 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-sys\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906527 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvkm\" (UniqueName: \"kubernetes.io/projected/b7caf5ef-960a-41c9-9b50-872641002718-kube-api-access-8zvkm\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-textfile\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-tls\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906637 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-wtmp\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.906689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907031 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907031 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906739 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-metrics-client-ca\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907031 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-wtmp\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907031 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.906960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-textfile\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907307 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.907283 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.907407 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.907388 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7caf5ef-960a-41c9-9b50-872641002718-metrics-client-ca\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.909700 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.909675 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.909807 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.909681 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7caf5ef-960a-41c9-9b50-872641002718-node-exporter-tls\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:28.914515 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:28.914492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvkm\" (UniqueName: \"kubernetes.io/projected/b7caf5ef-960a-41c9-9b50-872641002718-kube-api-access-8zvkm\") pod \"node-exporter-x9hh7\" (UID: \"b7caf5ef-960a-41c9-9b50-872641002718\") " pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:29.004284 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:29.004202 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9hh7" Jul 10 21:22:29.016028 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:22:29.015998 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7caf5ef_960a_41c9_9b50_872641002718.slice/crio-66964592facfa043ee2caf6711fc7c23ea05e32f23ea5ceeb27d6a66ad2a460e WatchSource:0}: Error finding container 66964592facfa043ee2caf6711fc7c23ea05e32f23ea5ceeb27d6a66ad2a460e: Status 404 returned error can't find the container with id 66964592facfa043ee2caf6711fc7c23ea05e32f23ea5ceeb27d6a66ad2a460e Jul 10 21:22:29.119915 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:29.119877 2561 patch_prober.go:28] interesting pod/image-registry-66b5c586bd-5d9xg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 21:22:29.120133 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:29.119947 2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" podUID="db491d18-185b-417c-a213-b80ddb40c74b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 21:22:29.770169 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:29.770123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9hh7" event={"ID":"b7caf5ef-960a-41c9-9b50-872641002718","Type":"ContainerStarted","Data":"66964592facfa043ee2caf6711fc7c23ea05e32f23ea5ceeb27d6a66ad2a460e"} Jul 10 21:22:30.776168 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:30.776133 2561 generic.go:356] "Generic (PLEG): container finished" podID="b7caf5ef-960a-41c9-9b50-872641002718" containerID="d64ae842c0ce9dbbd12e9d9c26aa26c104d7887196588f5f7d84d206dff1e066" exitCode=0 Jul 10 21:22:30.776658 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:30.776178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9hh7" event={"ID":"b7caf5ef-960a-41c9-9b50-872641002718","Type":"ContainerDied","Data":"d64ae842c0ce9dbbd12e9d9c26aa26c104d7887196588f5f7d84d206dff1e066"} Jul 10 21:22:31.617228 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:31.617199 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:22:31.782508 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:31.782462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9hh7" event={"ID":"b7caf5ef-960a-41c9-9b50-872641002718","Type":"ContainerStarted","Data":"cc8bcb0335170ad659d3b9e0871bd16a7a705d36df585b7b6aad3cfa2ef4cc07"} Jul 10 21:22:31.782904 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:31.782531 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9hh7" event={"ID":"b7caf5ef-960a-41c9-9b50-872641002718","Type":"ContainerStarted","Data":"61a54b4c4e559e96100997a49e10649be911928e77f144931927539ad22d4fa7"} Jul 10 21:22:31.803852 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:31.803799 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x9hh7" podStartSLOduration=2.796427451 podStartE2EDuration="3.803782416s" totalImagesPullingTime="931.359301ms" totalInitContainerRuntime="75.995664ms" isStatefulPod=true podCreationTimestamp="2026-07-10 21:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:22:31.802138172 +0000 UTC m=+90.152661769" watchObservedRunningTime="2026-07-10 21:22:31.803782416 +0000 UTC m=+90.154306051" Jul 10 21:22:35.171642 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:35.171595 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:22:41.615408 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:41.615379 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gqlw" Jul 10 21:22:53.865186 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:53.865146 2561 generic.go:356] "Generic (PLEG): container finished" podID="9781f26e-f121-4bdf-a5df-6507501cf548" containerID="354be0eff465a3d2fe6f47142efce120b238ffc67a2d4871ba8ccb0f193b4b73" exitCode=0 Jul 10 21:22:53.865606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:53.865224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" event={"ID":"9781f26e-f121-4bdf-a5df-6507501cf548","Type":"ContainerDied","Data":"354be0eff465a3d2fe6f47142efce120b238ffc67a2d4871ba8ccb0f193b4b73"} Jul 10 21:22:53.865606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:53.865563 2561 scope.go:117] "RemoveContainer" containerID="354be0eff465a3d2fe6f47142efce120b238ffc67a2d4871ba8ccb0f193b4b73" Jul 10 21:22:54.871570 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:54.871534 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-5hgv9" event={"ID":"9781f26e-f121-4bdf-a5df-6507501cf548","Type":"ContainerStarted","Data":"d23c28e9711c95b982b02c833ff7478e6f76e087a63f1255dd013358ab46c590"} Jul 10 21:22:58.887750 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:58.887716 2561 generic.go:356] "Generic (PLEG): container finished" podID="de2aecbb-9a62-48c4-887b-92b03ac8392b" containerID="47797b071df080fc48d6dd2f7558dd6524ec3cc3c37f217caa8d72caf19d26ec" exitCode=0 Jul 10 21:22:58.888214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:58.887786 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" event={"ID":"de2aecbb-9a62-48c4-887b-92b03ac8392b","Type":"ContainerDied","Data":"47797b071df080fc48d6dd2f7558dd6524ec3cc3c37f217caa8d72caf19d26ec"} Jul 10 21:22:58.888214 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:58.888165 2561 scope.go:117] "RemoveContainer" containerID="47797b071df080fc48d6dd2f7558dd6524ec3cc3c37f217caa8d72caf19d26ec" Jul 10 21:22:59.893704 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:22:59.893665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qqx7l" event={"ID":"de2aecbb-9a62-48c4-887b-92b03ac8392b","Type":"ContainerStarted","Data":"6215e454d8dffbba4ee16cd78ff66e14108df9c19a13f2f0a26a2ebc3f0653a3"} Jul 10 21:23:00.192930 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.192845 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" podUID="db491d18-185b-417c-a213-b80ddb40c74b" containerName="registry" containerID="cri-o://00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63" gracePeriod=30 Jul 10 21:23:00.444575 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.444518 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:23:00.562287 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562253 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562308 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562334 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562358 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562374 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562411 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562428 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn9pr\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562464 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562456 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration\") pod \"db491d18-185b-417c-a213-b80ddb40c74b\" (UID: \"db491d18-185b-417c-a213-b80ddb40c74b\") " Jul 10 21:23:00.562831 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562811 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:23:00.562943 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.562913 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:23:00.565090 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.565039 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:23:00.565311 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.565279 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:23:00.565419 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.565385 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr" (OuterVolumeSpecName: "kube-api-access-mn9pr") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "kube-api-access-mn9pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:23:00.565496 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.565472 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:23:00.565648 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.565632 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:23:00.571602 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.571569 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "db491d18-185b-417c-a213-b80ddb40c74b" (UID: "db491d18-185b-417c-a213-b80ddb40c74b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:23:00.663196 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663157 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-bound-sa-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663196 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663191 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9pr\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-kube-api-access-mn9pr\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663207 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-image-registry-private-configuration\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663222 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db491d18-185b-417c-a213-b80ddb40c74b-registry-tls\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663233 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-trusted-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663244 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db491d18-185b-417c-a213-b80ddb40c74b-ca-trust-extracted\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663255 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db491d18-185b-417c-a213-b80ddb40c74b-registry-certificates\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.663411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.663266 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db491d18-185b-417c-a213-b80ddb40c74b-installation-pull-secrets\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:23:00.898158 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.898116 2561 generic.go:356] "Generic (PLEG): container finished" podID="db491d18-185b-417c-a213-b80ddb40c74b" containerID="00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63" exitCode=0 Jul 10 21:23:00.898158 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.898160 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" Jul 10 21:23:00.898663 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.898203 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" event={"ID":"db491d18-185b-417c-a213-b80ddb40c74b","Type":"ContainerDied","Data":"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63"} Jul 10 21:23:00.898663 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.898256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b5c586bd-5d9xg" event={"ID":"db491d18-185b-417c-a213-b80ddb40c74b","Type":"ContainerDied","Data":"1b92c6a9bdf352ae2060c94efd85624083ab4a2ad64de51384d4bb541ad51b68"} Jul 10 21:23:00.898663 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.898280 2561 scope.go:117] "RemoveContainer" containerID="00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63" Jul 10 21:23:00.911964 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.911942 2561 scope.go:117] "RemoveContainer" containerID="00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63" Jul 10 21:23:00.912312 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:23:00.912286 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63\": container with ID starting with 00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63 not found: ID does not exist" containerID="00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63" Jul 10 21:23:00.912366 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.912322 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63"} err="failed to get container status \"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63\": rpc error: code = NotFound desc = could not find container \"00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63\": container with ID starting with 00c2206cfa9e368a90700b390941ab58bbb0c02785a293b11208615cfc392f63 not found: ID does not exist" Jul 10 21:23:00.923184 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.923146 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:23:00.927126 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:00.927103 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66b5c586bd-5d9xg"] Jul 10 21:23:02.182695 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:02.182580 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db491d18-185b-417c-a213-b80ddb40c74b" path="/var/lib/kubelet/pods/db491d18-185b-417c-a213-b80ddb40c74b/volumes" Jul 10 21:23:07.927671 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:07.927628 2561 generic.go:356] "Generic (PLEG): container finished" podID="68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d" containerID="ad10de6c7874a39b91fcf9df71570a00c66921f415e265dd88f4e0b1dda409ff" exitCode=0 Jul 10 21:23:07.928039 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:07.927697 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" event={"ID":"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d","Type":"ContainerDied","Data":"ad10de6c7874a39b91fcf9df71570a00c66921f415e265dd88f4e0b1dda409ff"} Jul 10 21:23:07.928039 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:07.928021 2561 scope.go:117] "RemoveContainer" containerID="ad10de6c7874a39b91fcf9df71570a00c66921f415e265dd88f4e0b1dda409ff" Jul 10 21:23:08.933364 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:23:08.933330 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-nz82z" event={"ID":"68dc5cd2-f5b4-41a7-abae-82f9ec7ce79d","Type":"ContainerStarted","Data":"764e33311a67032f37e54af2ff5a883c6cb51b8e2b7a72bc4be2100d589f659d"} Jul 10 21:25:22.093362 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.093272 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8zvm"] Jul 10 21:25:22.093853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.093564 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db491d18-185b-417c-a213-b80ddb40c74b" containerName="registry" Jul 10 21:25:22.093853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.093573 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db491d18-185b-417c-a213-b80ddb40c74b" containerName="registry" Jul 10 21:25:22.093853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.093620 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db491d18-185b-417c-a213-b80ddb40c74b" containerName="registry" Jul 10 21:25:22.095488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.095471 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.100267 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.100236 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5cxgl\"" Jul 10 21:25:22.100398 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.100239 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 21:25:22.100659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.100632 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 21:25:22.109231 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.109203 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8zvm"] Jul 10 21:25:22.177634 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.177588 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489vt\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-kube-api-access-489vt\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.177850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.177651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.278378 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.278333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489vt\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-kube-api-access-489vt\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.278561 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.278387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.286423 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.286395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.286653 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.286629 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-489vt\" (UniqueName: \"kubernetes.io/projected/87e68384-8eeb-4934-b77d-ed18045eb0dd-kube-api-access-489vt\") pod \"cert-manager-webhook-6486ffd997-l8zvm\" (UID: \"87e68384-8eeb-4934-b77d-ed18045eb0dd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.419194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.419157 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:22.560790 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:22.560762 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-l8zvm"] Jul 10 21:25:22.563314 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:25:22.563287 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87e68384_8eeb_4934_b77d_ed18045eb0dd.slice/crio-0832c60e7db53c883726bed04eb46a631aee494d5c8885ddec5f5b300dbd7b56 WatchSource:0}: Error finding container 0832c60e7db53c883726bed04eb46a631aee494d5c8885ddec5f5b300dbd7b56: Status 404 returned error can't find the container with id 0832c60e7db53c883726bed04eb46a631aee494d5c8885ddec5f5b300dbd7b56 Jul 10 21:25:23.449291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:23.449252 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" event={"ID":"87e68384-8eeb-4934-b77d-ed18045eb0dd","Type":"ContainerStarted","Data":"0832c60e7db53c883726bed04eb46a631aee494d5c8885ddec5f5b300dbd7b56"} Jul 10 21:25:26.466969 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:26.466924 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" event={"ID":"87e68384-8eeb-4934-b77d-ed18045eb0dd","Type":"ContainerStarted","Data":"ef561189b51492219c18cb4f3f49fd0114ff88b8f269e04cd3625421f9c43cb8"} Jul 10 21:25:26.467428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:26.467001 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:26.486231 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:26.486179 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" podStartSLOduration=1.495937821 podStartE2EDuration="4.486162142s" totalImagesPullingTime="2.990224321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:25:26.484487253 +0000 UTC m=+264.835010832" watchObservedRunningTime="2026-07-10 21:25:26.486162142 +0000 UTC m=+264.836685719" Jul 10 21:25:32.472987 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:32.472943 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-l8zvm" Jul 10 21:25:48.115839 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.115801 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp"] Jul 10 21:25:48.120530 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.120508 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.125025 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.124991 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 21:25:48.125236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.125146 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8jfkq\"" Jul 10 21:25:48.125340 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.125305 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 21:25:48.125396 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.125369 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 21:25:48.125843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.125826 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 21:25:48.143184 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.143150 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp"] Jul 10 21:25:48.194161 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.194121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.194339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.194191 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-manifests\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.194339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.194213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.194339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.194230 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-tmp\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.194339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.194308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55hz\" (UniqueName: \"kubernetes.io/projected/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-kube-api-access-q55hz\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.294767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.294729 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.294963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.294783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-manifests\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.294963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.294802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.294963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.294819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-tmp\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.294963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.294850 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55hz\" (UniqueName: \"kubernetes.io/projected/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-kube-api-access-q55hz\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.295333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.295302 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-manifests\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.297379 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.297344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-tmp\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.297521 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.297501 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.297583 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.297539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.312700 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.312671 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55hz\" (UniqueName: \"kubernetes.io/projected/1ca37d00-d45d-4fb3-9d26-5630b7d6aece-kube-api-access-q55hz\") pod \"opendatahub-operator-controller-manager-6d8977cb58-5l9vp\" (UID: \"1ca37d00-d45d-4fb3-9d26-5630b7d6aece\") " pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.430863 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.430760 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:48.495640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.495581 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-546985876b-8jr5b"] Jul 10 21:25:48.499412 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.499373 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.502659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502235 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 21:25:48.502659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502364 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 21:25:48.502659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502406 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 21:25:48.502659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502523 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 21:25:48.502659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502550 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rsfjz\"" Jul 10 21:25:48.502947 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.502930 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 21:25:48.513791 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.513765 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-546985876b-8jr5b"] Jul 10 21:25:48.597354 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.597311 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2d7j\" (UniqueName: \"kubernetes.io/projected/ea106490-e54e-4ba7-ab0a-7559bec3cb28-kube-api-access-t2d7j\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.597537 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.597370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.597537 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.597499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-metrics-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.597610 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.597538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea106490-e54e-4ba7-ab0a-7559bec3cb28-manager-config\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.613015 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.612981 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp"] Jul 10 21:25:48.615825 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:25:48.615794 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca37d00_d45d_4fb3_9d26_5630b7d6aece.slice/crio-d6a33efb5d53215c6b2ddc31ac21a6c65972c5937c88b064e4f4c34f58ed9040 WatchSource:0}: Error finding container d6a33efb5d53215c6b2ddc31ac21a6c65972c5937c88b064e4f4c34f58ed9040: Status 404 returned error can't find the container with id d6a33efb5d53215c6b2ddc31ac21a6c65972c5937c88b064e4f4c34f58ed9040 Jul 10 21:25:48.698905 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.698802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-metrics-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.698905 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.698848 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea106490-e54e-4ba7-ab0a-7559bec3cb28-manager-config\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.699159 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.698982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2d7j\" (UniqueName: \"kubernetes.io/projected/ea106490-e54e-4ba7-ab0a-7559bec3cb28-kube-api-access-t2d7j\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.699159 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.699041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.699647 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.699622 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea106490-e54e-4ba7-ab0a-7559bec3cb28-manager-config\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.701620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.701598 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-metrics-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.701709 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.701628 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea106490-e54e-4ba7-ab0a-7559bec3cb28-cert\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.713206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.713177 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2d7j\" (UniqueName: \"kubernetes.io/projected/ea106490-e54e-4ba7-ab0a-7559bec3cb28-kube-api-access-t2d7j\") pod \"lws-controller-manager-546985876b-8jr5b\" (UID: \"ea106490-e54e-4ba7-ab0a-7559bec3cb28\") " pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.810340 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.810296 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:48.963877 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:48.963831 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-546985876b-8jr5b"] Jul 10 21:25:48.965964 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:25:48.965937 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea106490_e54e_4ba7_ab0a_7559bec3cb28.slice/crio-4296587acedb8d3ebadf3eddc55e49798da882c9ce567cafaa46a5c2926a952a WatchSource:0}: Error finding container 4296587acedb8d3ebadf3eddc55e49798da882c9ce567cafaa46a5c2926a952a: Status 404 returned error can't find the container with id 4296587acedb8d3ebadf3eddc55e49798da882c9ce567cafaa46a5c2926a952a Jul 10 21:25:49.559742 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:49.559702 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" event={"ID":"1ca37d00-d45d-4fb3-9d26-5630b7d6aece","Type":"ContainerStarted","Data":"d6a33efb5d53215c6b2ddc31ac21a6c65972c5937c88b064e4f4c34f58ed9040"} Jul 10 21:25:49.561188 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:49.561144 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" event={"ID":"ea106490-e54e-4ba7-ab0a-7559bec3cb28","Type":"ContainerStarted","Data":"4296587acedb8d3ebadf3eddc55e49798da882c9ce567cafaa46a5c2926a952a"} Jul 10 21:25:53.589902 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:53.589867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" event={"ID":"1ca37d00-d45d-4fb3-9d26-5630b7d6aece","Type":"ContainerStarted","Data":"0892a2bd1820449ecb1992455419b5df029ac773ce3e6f61a656328624c15b64"} Jul 10 21:25:53.591429 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:53.591376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" event={"ID":"ea106490-e54e-4ba7-ab0a-7559bec3cb28","Type":"ContainerStarted","Data":"bb729b4d1e4fec4324868ddaf5d039a603dcb7b500164908736d8cf53d11df3d"} Jul 10 21:25:53.591562 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:53.591507 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:25:53.631259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:53.631199 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" podStartSLOduration=1.621737137 podStartE2EDuration="5.631178712s" totalImagesPullingTime="4.009441575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:25:53.630256506 +0000 UTC m=+291.980780103" watchObservedRunningTime="2026-07-10 21:25:53.631178712 +0000 UTC m=+291.981702289" Jul 10 21:25:54.596828 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:54.596789 2561 generic.go:356] "Generic (PLEG): container finished" podID="1ca37d00-d45d-4fb3-9d26-5630b7d6aece" containerID="0892a2bd1820449ecb1992455419b5df029ac773ce3e6f61a656328624c15b64" exitCode=0 Jul 10 21:25:54.597260 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:54.596882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" event={"ID":"1ca37d00-d45d-4fb3-9d26-5630b7d6aece","Type":"ContainerDied","Data":"0892a2bd1820449ecb1992455419b5df029ac773ce3e6f61a656328624c15b64"} Jul 10 21:25:55.602782 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:55.602744 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" event={"ID":"1ca37d00-d45d-4fb3-9d26-5630b7d6aece","Type":"ContainerStarted","Data":"d219039d5f8290f6f63673ed1b45471437501f54769df62d134473282e611b1e"} Jul 10 21:25:55.603257 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:55.602867 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:25:55.629126 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:25:55.629043 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" podStartSLOduration=1.8876259709999998 podStartE2EDuration="7.629026093s" totalImagesPullingTime="4.724252732s" totalInitContainerRuntime="1.01714739s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:25:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:25:55.626933315 +0000 UTC m=+293.977456928" watchObservedRunningTime="2026-07-10 21:25:55.629026093 +0000 UTC m=+293.979549746" Jul 10 21:26:02.052046 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:02.052015 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:26:02.055498 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:02.055471 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:26:02.060250 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:02.060223 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:26:02.063463 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:02.063439 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:26:02.063607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:02.063444 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 21:26:04.600267 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:04.600237 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-546985876b-8jr5b" Jul 10 21:26:06.609318 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:06.609287 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d8977cb58-5l9vp" Jul 10 21:26:17.999256 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:17.999213 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:18.002630 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.002612 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.006246 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.006212 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 21:26:18.006387 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.006289 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8t7bh\"" Jul 10 21:26:18.016436 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.016407 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:18.044555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044754 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044754 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044671 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044754 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.044948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.044846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdls\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.145876 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.145832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.145876 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.145874 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.145923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.145960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.145989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146030 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdls\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146527 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146394 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146527 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146490 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146639 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146536 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146639 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146622 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.146982 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.146957 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.148447 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.148426 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.148783 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.148764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.156319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.156283 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdls\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.156580 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.156560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-vtx65\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.241000 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.240960 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7"] Jul 10 21:26:18.245381 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.245360 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.280099 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.280043 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7"] Jul 10 21:26:18.313197 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.313163 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:18.347799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.347799 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347802 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347923 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df4b9d9-8f7f-4211-b328-e61375820b42-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347953 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.347971 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfcz\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-kube-api-access-jqfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.348014 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.348100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.348058 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449219 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449177 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449219 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449238 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df4b9d9-8f7f-4211-b328-e61375820b42-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449351 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449377 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfcz\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-kube-api-access-jqfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449739 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.449996 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.449968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.450271 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.450254 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df4b9d9-8f7f-4211-b328-e61375820b42-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.451780 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.451761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.451996 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.451980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.471038 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.471008 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfcz\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-kube-api-access-jqfcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.474743 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.474711 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df4b9d9-8f7f-4211-b328-e61375820b42-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vt9b7\" (UID: \"8df4b9d9-8f7f-4211-b328-e61375820b42\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.479851 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.479823 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:18.483055 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:26:18.483027 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e0f339_ae72_4dc6_bc8f_4f522aaf5fb2.slice/crio-c92657d2a9c3be7614461110099338d4b96a850c8249516d8e370eb2684623d6 WatchSource:0}: Error finding container c92657d2a9c3be7614461110099338d4b96a850c8249516d8e370eb2684623d6: Status 404 returned error can't find the container with id c92657d2a9c3be7614461110099338d4b96a850c8249516d8e370eb2684623d6 Jul 10 21:26:18.485139 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.485122 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:26:18.556569 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.556467 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:18.703877 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.703850 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7"] Jul 10 21:26:18.705880 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:26:18.705849 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df4b9d9_8f7f_4211_b328_e61375820b42.slice/crio-509209dc2875750888f792edd70abc162100059e95c8a70dc455d4da4d1d3b0a WatchSource:0}: Error finding container 509209dc2875750888f792edd70abc162100059e95c8a70dc455d4da4d1d3b0a: Status 404 returned error can't find the container with id 509209dc2875750888f792edd70abc162100059e95c8a70dc455d4da4d1d3b0a Jul 10 21:26:18.710943 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.710917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" event={"ID":"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2","Type":"ContainerStarted","Data":"c92657d2a9c3be7614461110099338d4b96a850c8249516d8e370eb2684623d6"} Jul 10 21:26:18.767413 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.767379 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7"] Jul 10 21:26:18.772029 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.772008 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.776551 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.776523 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 21:26:18.776551 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.776547 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 21:26:18.776771 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.776588 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kl5kn\"" Jul 10 21:26:18.776771 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.776531 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 21:26:18.788423 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.788393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7"] Jul 10 21:26:18.853134 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.852965 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfth\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-kube-api-access-kwfth\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853134 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853134 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853058 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.853416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.853289 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954078 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954035 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954194 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfth\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-kube-api-access-kwfth\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954290 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.954837 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.954814 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.956718 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.956695 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.956821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.956764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.957126 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.957107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.957342 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.957323 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.965621 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.965591 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfth\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-kube-api-access-kwfth\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:18.968180 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:18.968159 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b9fa2c-57db-4859-bd54-72efa3c9a00d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7mfz7\" (UID: \"d6b9fa2c-57db-4859-bd54-72efa3c9a00d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:19.082991 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:19.082953 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:19.241174 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:19.241139 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7"] Jul 10 21:26:19.245192 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:26:19.245156 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9fa2c_57db_4859_bd54_72efa3c9a00d.slice/crio-6495056cb384d883332206d837ba4303d70a91ce40b7e0b76d45c8c127eb2ab4 WatchSource:0}: Error finding container 6495056cb384d883332206d837ba4303d70a91ce40b7e0b76d45c8c127eb2ab4: Status 404 returned error can't find the container with id 6495056cb384d883332206d837ba4303d70a91ce40b7e0b76d45c8c127eb2ab4 Jul 10 21:26:19.720608 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:19.720569 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" event={"ID":"d6b9fa2c-57db-4859-bd54-72efa3c9a00d","Type":"ContainerStarted","Data":"6495056cb384d883332206d837ba4303d70a91ce40b7e0b76d45c8c127eb2ab4"} Jul 10 21:26:19.722881 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:19.722847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" event={"ID":"8df4b9d9-8f7f-4211-b328-e61375820b42","Type":"ContainerStarted","Data":"509209dc2875750888f792edd70abc162100059e95c8a70dc455d4da4d1d3b0a"} Jul 10 21:26:22.457281 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.457222 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.457631 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.457324 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.457631 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.457221 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.457631 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.457402 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.457631 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.457436 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.464672 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.464640 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.464766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.464709 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.464766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.464738 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:26:22.741265 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.741172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" event={"ID":"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2","Type":"ContainerStarted","Data":"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2"} Jul 10 21:26:22.743019 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.742988 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" event={"ID":"d6b9fa2c-57db-4859-bd54-72efa3c9a00d","Type":"ContainerStarted","Data":"3a70d3113f7c9d4640d8eb27f5bfae1e3da7e7623ecb5ada57c649e8eea144a5"} Jul 10 21:26:22.743157 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.743118 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:22.744628 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.744604 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" event={"ID":"8df4b9d9-8f7f-4211-b328-e61375820b42","Type":"ContainerStarted","Data":"0dd275f509d5a15191706af9f4db2841263b11864e1c8b0449b488e56c62028e"} Jul 10 21:26:22.766021 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.765953 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" podStartSLOduration=1.794295558 podStartE2EDuration="5.765933485s" totalImagesPullingTime="3.971637927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:26:22.764674335 +0000 UTC m=+321.115197937" watchObservedRunningTime="2026-07-10 21:26:22.765933485 +0000 UTC m=+321.116457064" Jul 10 21:26:22.789838 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.789776 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" podStartSLOduration=1.581737166 podStartE2EDuration="4.789750992s" totalImagesPullingTime="3.208013826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:26:22.786951161 +0000 UTC m=+321.137474772" watchObservedRunningTime="2026-07-10 21:26:22.789750992 +0000 UTC m=+321.140274572" Jul 10 21:26:22.811794 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:22.811735 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" podStartSLOduration=1.055476959 podStartE2EDuration="4.811713195s" totalImagesPullingTime="3.756236236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:26:22.811240532 +0000 UTC m=+321.161764129" watchObservedRunningTime="2026-07-10 21:26:22.811713195 +0000 UTC m=+321.162236775" Jul 10 21:26:23.313792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.313748 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:23.315341 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.315313 2561 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 10 21:26:23.315497 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.315370 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 10 21:26:23.557451 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.557405 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:23.562243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.562218 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:23.749253 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.749217 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:23.750132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.750104 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vt9b7" Jul 10 21:26:23.751081 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.751046 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7mfz7" Jul 10 21:26:23.810265 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:23.810228 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:24.314003 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:24.313963 2561 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 10 21:26:24.314207 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:24.314029 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 10 21:26:25.313564 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:25.313525 2561 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 10 21:26:25.314021 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:25.313582 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 10 21:26:25.758042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:25.757978 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" containerID="cri-o://18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2" gracePeriod=30 Jul 10 21:26:31.014430 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.014402 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:31.061482 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061396 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061482 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061443 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061482 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061484 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061582 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061644 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbdls\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061683 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061710 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061727 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.061767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061751 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061764 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061763 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert\") pod \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\" (UID: \"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2\") " Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061968 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.061987 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.062101 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.062325 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.062346 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.062360 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.062611 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.062373 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.063909 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.063881 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:26:31.064007 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.063907 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:26:31.064100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.064023 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:26:31.064146 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.064114 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls" (OuterVolumeSpecName: "kube-api-access-wbdls") pod "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" (UID: "11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2"). InnerVolumeSpecName "kube-api-access-wbdls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:26:31.163685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.163635 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.163685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.163674 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.163685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.163684 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbdls\" (UniqueName: \"kubernetes.io/projected/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-kube-api-access-wbdls\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.163685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.163694 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.163957 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.163702 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:26:31.785347 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.785309 2561 generic.go:356] "Generic (PLEG): container finished" podID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerID="18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2" exitCode=0 Jul 10 21:26:31.785519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.785361 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" Jul 10 21:26:31.785519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.785368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" event={"ID":"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2","Type":"ContainerDied","Data":"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2"} Jul 10 21:26:31.785519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.785417 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65" event={"ID":"11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2","Type":"ContainerDied","Data":"c92657d2a9c3be7614461110099338d4b96a850c8249516d8e370eb2684623d6"} Jul 10 21:26:31.785519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.785439 2561 scope.go:117] "RemoveContainer" containerID="18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2" Jul 10 21:26:31.794719 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.794701 2561 scope.go:117] "RemoveContainer" containerID="18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2" Jul 10 21:26:31.794996 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:26:31.794976 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2\": container with ID starting with 18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2 not found: ID does not exist" containerID="18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2" Jul 10 21:26:31.795042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.795008 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2"} err="failed to get container status \"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2\": rpc error: code = NotFound desc = could not find container \"18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2\": container with ID starting with 18567556d8c6be1b1a71a8b68e372234304b8424f5f7b4b0bb9628fa6513b0c2 not found: ID does not exist" Jul 10 21:26:31.808169 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.808137 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:31.812685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:31.812649 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-vtx65"] Jul 10 21:26:32.184948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:26:32.184899 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" path="/var/lib/kubelet/pods/11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2/volumes" Jul 10 21:28:55.535635 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.535597 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:28:55.536132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.535909 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" Jul 10 21:28:55.536132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.535921 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" Jul 10 21:28:55.536132 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.535993 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e0f339-ae72-4dc6-bc8f-4f522aaf5fb2" containerName="istio-proxy" Jul 10 21:28:55.538101 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.538056 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:28:55.540503 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.540473 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ljm7t\"" Jul 10 21:28:55.540634 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.540463 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 21:28:55.540634 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.540530 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 21:28:55.547052 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.547026 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:28:55.635857 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.635816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjp8j\" (UniqueName: \"kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j\") pod \"authorino-58cd955675-n5tgk\" (UID: \"766a77ad-aca8-40ad-bd22-7b1c32621f0c\") " pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:28:55.736951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.736909 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjp8j\" (UniqueName: \"kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j\") pod \"authorino-58cd955675-n5tgk\" (UID: \"766a77ad-aca8-40ad-bd22-7b1c32621f0c\") " pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:28:55.746875 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.746848 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjp8j\" (UniqueName: \"kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j\") pod \"authorino-58cd955675-n5tgk\" (UID: \"766a77ad-aca8-40ad-bd22-7b1c32621f0c\") " pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:28:55.848843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.848747 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:28:55.992449 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:55.992423 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:28:55.995111 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:28:55.995079 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod766a77ad_aca8_40ad_bd22_7b1c32621f0c.slice/crio-1275abc965d4e4d6152ababf9b8b7e08efec9928c18f179541372c268245057b WatchSource:0}: Error finding container 1275abc965d4e4d6152ababf9b8b7e08efec9928c18f179541372c268245057b: Status 404 returned error can't find the container with id 1275abc965d4e4d6152ababf9b8b7e08efec9928c18f179541372c268245057b Jul 10 21:28:56.421273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:56.421240 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n5tgk" event={"ID":"766a77ad-aca8-40ad-bd22-7b1c32621f0c","Type":"ContainerStarted","Data":"1275abc965d4e4d6152ababf9b8b7e08efec9928c18f179541372c268245057b"} Jul 10 21:28:59.437936 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:59.437845 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n5tgk" event={"ID":"766a77ad-aca8-40ad-bd22-7b1c32621f0c","Type":"ContainerStarted","Data":"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657"} Jul 10 21:28:59.454019 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:28:59.453963 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-n5tgk" podStartSLOduration=1.3346737929999999 podStartE2EDuration="4.453946145s" totalImagesPullingTime="3.119272352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:28:59.45275349 +0000 UTC m=+477.803277071" watchObservedRunningTime="2026-07-10 21:28:59.453946145 +0000 UTC m=+477.804469723" Jul 10 21:29:00.190834 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:00.190800 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:29:01.446667 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:01.446623 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-n5tgk" podUID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" containerName="authorino" containerID="cri-o://4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657" gracePeriod=30 Jul 10 21:29:01.695244 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:01.695212 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:29:01.785102 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:01.784972 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjp8j\" (UniqueName: \"kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j\") pod \"766a77ad-aca8-40ad-bd22-7b1c32621f0c\" (UID: \"766a77ad-aca8-40ad-bd22-7b1c32621f0c\") " Jul 10 21:29:01.787374 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:01.787345 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j" (OuterVolumeSpecName: "kube-api-access-qjp8j") pod "766a77ad-aca8-40ad-bd22-7b1c32621f0c" (UID: "766a77ad-aca8-40ad-bd22-7b1c32621f0c"). InnerVolumeSpecName "kube-api-access-qjp8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:29:01.886384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:01.886343 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjp8j\" (UniqueName: \"kubernetes.io/projected/766a77ad-aca8-40ad-bd22-7b1c32621f0c-kube-api-access-qjp8j\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:29:02.452020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.451915 2561 generic.go:356] "Generic (PLEG): container finished" podID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" containerID="4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657" exitCode=0 Jul 10 21:29:02.452020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.451951 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n5tgk" Jul 10 21:29:02.452020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.452004 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n5tgk" event={"ID":"766a77ad-aca8-40ad-bd22-7b1c32621f0c","Type":"ContainerDied","Data":"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657"} Jul 10 21:29:02.452529 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.452045 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n5tgk" event={"ID":"766a77ad-aca8-40ad-bd22-7b1c32621f0c","Type":"ContainerDied","Data":"1275abc965d4e4d6152ababf9b8b7e08efec9928c18f179541372c268245057b"} Jul 10 21:29:02.452529 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.452084 2561 scope.go:117] "RemoveContainer" containerID="4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657" Jul 10 21:29:02.460637 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.460616 2561 scope.go:117] "RemoveContainer" containerID="4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657" Jul 10 21:29:02.460909 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:29:02.460890 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657\": container with ID starting with 4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657 not found: ID does not exist" containerID="4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657" Jul 10 21:29:02.460959 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.460917 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657"} err="failed to get container status \"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657\": rpc error: code = NotFound desc = could not find container \"4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657\": container with ID starting with 4b5b16d74a1a77f0246e2706dfc69bb0a82d3cd4e51a0b3120c89af4edfae657 not found: ID does not exist" Jul 10 21:29:02.468806 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.468776 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:29:02.474653 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:02.474627 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-n5tgk"] Jul 10 21:29:04.185414 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:04.185375 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" path="/var/lib/kubelet/pods/766a77ad-aca8-40ad-bd22-7b1c32621f0c/volumes" Jul 10 21:29:11.279660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.279617 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk"] Jul 10 21:29:11.280160 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.279927 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" containerName="authorino" Jul 10 21:29:11.280160 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.279937 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" containerName="authorino" Jul 10 21:29:11.280160 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.279997 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="766a77ad-aca8-40ad-bd22-7b1c32621f0c" containerName="authorino" Jul 10 21:29:11.283114 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.283090 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.285600 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.285574 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ptss7\"" Jul 10 21:29:11.285734 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.285586 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 21:29:11.286294 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.286276 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 21:29:11.286422 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.286299 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 21:29:11.297185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.297152 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk"] Jul 10 21:29:11.369128 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.369050 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cba446f-4f19-458c-8fc8-66bbd1306c71-data\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.369309 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.369225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhg2c\" (UniqueName: \"kubernetes.io/projected/3cba446f-4f19-458c-8fc8-66bbd1306c71-kube-api-access-fhg2c\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.469685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.469645 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cba446f-4f19-458c-8fc8-66bbd1306c71-data\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.469886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.469743 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhg2c\" (UniqueName: \"kubernetes.io/projected/3cba446f-4f19-458c-8fc8-66bbd1306c71-kube-api-access-fhg2c\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.470126 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.470104 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cba446f-4f19-458c-8fc8-66bbd1306c71-data\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.478559 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.478531 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhg2c\" (UniqueName: \"kubernetes.io/projected/3cba446f-4f19-458c-8fc8-66bbd1306c71-kube-api-access-fhg2c\") pod \"postgres-69fb4594cf-fz8hk\" (UID: \"3cba446f-4f19-458c-8fc8-66bbd1306c71\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.594960 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.594855 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:11.732912 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:11.732877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk"] Jul 10 21:29:11.735929 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:29:11.735901 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cba446f_4f19_458c_8fc8_66bbd1306c71.slice/crio-d7657b9d2d3585d34760c7c5a6e77da817ec9fdd5ad7dd5b3624a90f117802a2 WatchSource:0}: Error finding container d7657b9d2d3585d34760c7c5a6e77da817ec9fdd5ad7dd5b3624a90f117802a2: Status 404 returned error can't find the container with id d7657b9d2d3585d34760c7c5a6e77da817ec9fdd5ad7dd5b3624a90f117802a2 Jul 10 21:29:12.497845 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:12.497804 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" event={"ID":"3cba446f-4f19-458c-8fc8-66bbd1306c71","Type":"ContainerStarted","Data":"d7657b9d2d3585d34760c7c5a6e77da817ec9fdd5ad7dd5b3624a90f117802a2"} Jul 10 21:29:17.525520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:17.525482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" event={"ID":"3cba446f-4f19-458c-8fc8-66bbd1306c71","Type":"ContainerStarted","Data":"39948ee49779000f469b347a6a5147af0e6611b2b944b026517b0d7771725ba0"} Jul 10 21:29:17.525942 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:17.525609 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:17.543264 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:17.543211 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" podStartSLOduration=1.57643317 podStartE2EDuration="6.543194338s" totalImagesPullingTime="4.966761168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:29:17.541226072 +0000 UTC m=+495.891749664" watchObservedRunningTime="2026-07-10 21:29:17.543194338 +0000 UTC m=+495.893717916" Jul 10 21:29:23.561543 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:23.561508 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fz8hk" Jul 10 21:29:24.134343 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.134307 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kkrd5"] Jul 10 21:29:24.136756 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.136738 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.139524 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.139497 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 21:29:24.139978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.139960 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 21:29:24.140109 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.140033 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ljm7t\"" Jul 10 21:29:24.146763 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.146740 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kkrd5"] Jul 10 21:29:24.283821 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.283776 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977cj\" (UniqueName: \"kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj\") pod \"authorino-5d4b67c446-kkrd5\" (UID: \"6485a7f7-6894-4a61-a44d-377cd876ca3b\") " pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.369890 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.369855 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kkrd5"] Jul 10 21:29:24.370165 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:29:24.370144 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-977cj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-kkrd5" podUID="6485a7f7-6894-4a61-a44d-377cd876ca3b" Jul 10 21:29:24.384634 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.384560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977cj\" (UniqueName: \"kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj\") pod \"authorino-5d4b67c446-kkrd5\" (UID: \"6485a7f7-6894-4a61-a44d-377cd876ca3b\") " pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.403842 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.403813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-977cj\" (UniqueName: \"kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj\") pod \"authorino-5d4b67c446-kkrd5\" (UID: \"6485a7f7-6894-4a61-a44d-377cd876ca3b\") " pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.405400 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.405378 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d7ccc95ff-pxpn4"] Jul 10 21:29:24.408322 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.408302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:24.417085 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.417035 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d7ccc95ff-pxpn4"] Jul 10 21:29:24.485036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.485005 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstvq\" (UniqueName: \"kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq\") pod \"authorino-7d7ccc95ff-pxpn4\" (UID: \"a2cf04e1-e890-4b75-b4c5-e8b14791a22e\") " pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:24.557090 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.557032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.562958 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.562936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:24.585696 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.585653 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstvq\" (UniqueName: \"kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq\") pod \"authorino-7d7ccc95ff-pxpn4\" (UID: \"a2cf04e1-e890-4b75-b4c5-e8b14791a22e\") " pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:24.591346 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.591315 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7ccc95ff-pxpn4"] Jul 10 21:29:24.591590 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:29:24.591571 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pstvq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" podUID="a2cf04e1-e890-4b75-b4c5-e8b14791a22e" Jul 10 21:29:24.595470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.595444 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstvq\" (UniqueName: \"kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq\") pod \"authorino-7d7ccc95ff-pxpn4\" (UID: \"a2cf04e1-e890-4b75-b4c5-e8b14791a22e\") " pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:24.629047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.629008 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:29:24.631326 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.631309 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.634396 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.634374 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 21:29:24.649796 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.649761 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:29:24.686045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.686006 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-977cj\" (UniqueName: \"kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj\") pod \"6485a7f7-6894-4a61-a44d-377cd876ca3b\" (UID: \"6485a7f7-6894-4a61-a44d-377cd876ca3b\") " Jul 10 21:29:24.686273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.686195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm59w\" (UniqueName: \"kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.686338 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.686290 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.688456 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.688429 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj" (OuterVolumeSpecName: "kube-api-access-977cj") pod "6485a7f7-6894-4a61-a44d-377cd876ca3b" (UID: "6485a7f7-6894-4a61-a44d-377cd876ca3b"). InnerVolumeSpecName "kube-api-access-977cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:29:24.787081 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.787021 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.787254 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.787126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm59w\" (UniqueName: \"kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.787254 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.787197 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-977cj\" (UniqueName: \"kubernetes.io/projected/6485a7f7-6894-4a61-a44d-377cd876ca3b-kube-api-access-977cj\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:29:24.789740 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.789714 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.798586 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.798560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm59w\" (UniqueName: \"kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w\") pod \"authorino-54b465d987-cs7mt\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:24.941219 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:24.941129 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:29:25.080326 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.080295 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:29:25.082752 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:29:25.082720 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa1ea812_9aa8_4546_be22_d27ee389b3c2.slice/crio-1f93f5f3d8d96f3e6f70d52d9b4c76a11a74ca824251717263c2b054f540b4f0 WatchSource:0}: Error finding container 1f93f5f3d8d96f3e6f70d52d9b4c76a11a74ca824251717263c2b054f540b4f0: Status 404 returned error can't find the container with id 1f93f5f3d8d96f3e6f70d52d9b4c76a11a74ca824251717263c2b054f540b4f0 Jul 10 21:29:25.564385 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.564335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b465d987-cs7mt" event={"ID":"aa1ea812-9aa8-4546-be22-d27ee389b3c2","Type":"ContainerStarted","Data":"9429d0f2363718a2d9dcd424fdc1cf29106d81c957f0666016e9fecdc43df02d"} Jul 10 21:29:25.564385 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.564377 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:25.564948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.564419 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kkrd5" Jul 10 21:29:25.564948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.564390 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b465d987-cs7mt" event={"ID":"aa1ea812-9aa8-4546-be22-d27ee389b3c2","Type":"ContainerStarted","Data":"1f93f5f3d8d96f3e6f70d52d9b4c76a11a74ca824251717263c2b054f540b4f0"} Jul 10 21:29:25.571393 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.571358 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:25.587766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.587711 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54b465d987-cs7mt" podStartSLOduration=1.2456631 podStartE2EDuration="1.587692542s" totalImagesPullingTime="342.029442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:29:25.585160784 +0000 UTC m=+503.935684362" watchObservedRunningTime="2026-07-10 21:29:25.587692542 +0000 UTC m=+503.938216119" Jul 10 21:29:25.612795 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.612759 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kkrd5"] Jul 10 21:29:25.618978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.618941 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kkrd5"] Jul 10 21:29:25.696035 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.695943 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pstvq\" (UniqueName: \"kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq\") pod \"a2cf04e1-e890-4b75-b4c5-e8b14791a22e\" (UID: \"a2cf04e1-e890-4b75-b4c5-e8b14791a22e\") " Jul 10 21:29:25.698323 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.698289 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq" (OuterVolumeSpecName: "kube-api-access-pstvq") pod "a2cf04e1-e890-4b75-b4c5-e8b14791a22e" (UID: "a2cf04e1-e890-4b75-b4c5-e8b14791a22e"). InnerVolumeSpecName "kube-api-access-pstvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:29:25.797494 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:25.797450 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstvq\" (UniqueName: \"kubernetes.io/projected/a2cf04e1-e890-4b75-b4c5-e8b14791a22e-kube-api-access-pstvq\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:29:26.183822 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:26.183783 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6485a7f7-6894-4a61-a44d-377cd876ca3b" path="/var/lib/kubelet/pods/6485a7f7-6894-4a61-a44d-377cd876ca3b/volumes" Jul 10 21:29:26.568236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:26.568155 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7ccc95ff-pxpn4" Jul 10 21:29:26.600075 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:26.600031 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7ccc95ff-pxpn4"] Jul 10 21:29:26.604907 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:26.604877 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d7ccc95ff-pxpn4"] Jul 10 21:29:28.183744 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:29:28.183710 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2cf04e1-e890-4b75-b4c5-e8b14791a22e" path="/var/lib/kubelet/pods/a2cf04e1-e890-4b75-b4c5-e8b14791a22e/volumes" Jul 10 21:30:34.802136 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.802093 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx"] Jul 10 21:30:34.805812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.805785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.808819 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.808791 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 21:30:34.809679 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.809661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 21:30:34.809762 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.809702 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 21:30:34.809762 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.809703 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-crb2j\"" Jul 10 21:30:34.816196 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb4bae-7ce5-450b-aa5b-22319d499f83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.816291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816203 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.816291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816250 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.816291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816272 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jx4\" (UniqueName: \"kubernetes.io/projected/88bb4bae-7ce5-450b-aa5b-22319d499f83-kube-api-access-n4jx4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.816395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816291 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.816395 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.816318 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.819817 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.819792 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx"] Jul 10 21:30:34.917513 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb4bae-7ce5-450b-aa5b-22319d499f83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.917513 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.917766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917547 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.917766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jx4\" (UniqueName: \"kubernetes.io/projected/88bb4bae-7ce5-450b-aa5b-22319d499f83-kube-api-access-n4jx4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.917766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917604 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.917766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.917647 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.918105 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.918078 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.918405 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.918107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.918405 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.918164 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.920110 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.920048 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88bb4bae-7ce5-450b-aa5b-22319d499f83-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.920244 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.920226 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88bb4bae-7ce5-450b-aa5b-22319d499f83-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:34.927171 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:34.927148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jx4\" (UniqueName: \"kubernetes.io/projected/88bb4bae-7ce5-450b-aa5b-22319d499f83-kube-api-access-n4jx4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5gpx\" (UID: \"88bb4bae-7ce5-450b-aa5b-22319d499f83\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:35.116930 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:35.116894 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:35.260033 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:35.259996 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx"] Jul 10 21:30:35.263193 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:30:35.263164 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bb4bae_7ce5_450b_aa5b_22319d499f83.slice/crio-2598ce88378efbbd6234089cf096303d2c185699fe8ce3eb4956d4092c1d9fd6 WatchSource:0}: Error finding container 2598ce88378efbbd6234089cf096303d2c185699fe8ce3eb4956d4092c1d9fd6: Status 404 returned error can't find the container with id 2598ce88378efbbd6234089cf096303d2c185699fe8ce3eb4956d4092c1d9fd6 Jul 10 21:30:35.885330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:35.885243 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" event={"ID":"88bb4bae-7ce5-450b-aa5b-22319d499f83","Type":"ContainerStarted","Data":"2598ce88378efbbd6234089cf096303d2c185699fe8ce3eb4956d4092c1d9fd6"} Jul 10 21:30:36.893184 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:36.893151 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" event={"ID":"88bb4bae-7ce5-450b-aa5b-22319d499f83","Type":"ContainerStarted","Data":"e5090a8373a917b23009a84517bf2195627b738eae0b60a5939b58a733625ece"} Jul 10 21:30:36.916449 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:36.916391 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" podStartSLOduration=1.775604293 podStartE2EDuration="2.916370594s" totalImagesPullingTime="1.140766301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:30:36.914158809 +0000 UTC m=+575.264682403" watchObservedRunningTime="2026-07-10 21:30:36.916370594 +0000 UTC m=+575.266894172" Jul 10 21:30:45.117354 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:45.117300 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:45.117869 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:45.117386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:45.130047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:45.130016 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:45.944299 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:45.944259 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5gpx" Jul 10 21:30:52.020139 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.020098 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6"] Jul 10 21:30:52.023598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.023579 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.027019 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.026994 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 21:30:52.043087 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.042225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6"] Jul 10 21:30:52.074877 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.074842 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb6b\" (UniqueName: \"kubernetes.io/projected/bc7f4542-28f7-4ae6-88ca-e58132a0f015-kube-api-access-nvb6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.075058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.074887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.075058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.074914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.075058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.075014 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.075208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.075104 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.075208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.075160 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.175838 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.175794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.175867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb6b\" (UniqueName: \"kubernetes.io/projected/bc7f4542-28f7-4ae6-88ca-e58132a0f015-kube-api-access-nvb6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.175892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.175919 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.175947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176290 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.176094 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.176365 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176462 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.176441 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.176502 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.176478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.178491 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.178459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7f4542-28f7-4ae6-88ca-e58132a0f015-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.178838 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.178813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f4542-28f7-4ae6-88ca-e58132a0f015-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.186743 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.186715 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb6b\" (UniqueName: \"kubernetes.io/projected/bc7f4542-28f7-4ae6-88ca-e58132a0f015-kube-api-access-nvb6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6\" (UID: \"bc7f4542-28f7-4ae6-88ca-e58132a0f015\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.334242 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.334143 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:30:52.482772 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.482743 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6"] Jul 10 21:30:52.485344 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:30:52.485317 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7f4542_28f7_4ae6_88ca_e58132a0f015.slice/crio-d7515c0a3e27646193c1b1016c5bca4b1ad211e874866f6b0376599e0be5fa07 WatchSource:0}: Error finding container d7515c0a3e27646193c1b1016c5bca4b1ad211e874866f6b0376599e0be5fa07: Status 404 returned error can't find the container with id d7515c0a3e27646193c1b1016c5bca4b1ad211e874866f6b0376599e0be5fa07 Jul 10 21:30:52.968039 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.967949 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" event={"ID":"bc7f4542-28f7-4ae6-88ca-e58132a0f015","Type":"ContainerStarted","Data":"3ea1f6d6b5548d2a3717d953a4ab88c15af426b99bafddb469367307d74ca63d"} Jul 10 21:30:52.968039 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.968001 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" event={"ID":"bc7f4542-28f7-4ae6-88ca-e58132a0f015","Type":"ContainerStarted","Data":"d7515c0a3e27646193c1b1016c5bca4b1ad211e874866f6b0376599e0be5fa07"} Jul 10 21:30:52.989660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:30:52.989608 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" podStartSLOduration=1.803058546 podStartE2EDuration="1.989591765s" totalImagesPullingTime="186.533219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:30:52.98853096 +0000 UTC m=+591.339054564" watchObservedRunningTime="2026-07-10 21:30:52.989591765 +0000 UTC m=+591.340115343" Jul 10 21:31:02.084353 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.084316 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:31:02.086345 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.086316 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:31:02.091922 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.091900 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:31:02.093917 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.093891 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:31:02.334831 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.334740 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:31:02.334996 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.334890 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:31:02.347501 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:02.347460 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:31:03.024949 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:03.024914 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6" Jul 10 21:31:38.030300 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.030204 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:31:38.032853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.030434 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-54b465d987-cs7mt" podUID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" containerName="authorino" containerID="cri-o://9429d0f2363718a2d9dcd424fdc1cf29106d81c957f0666016e9fecdc43df02d" gracePeriod=30 Jul 10 21:31:38.181324 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.181282 2561 generic.go:356] "Generic (PLEG): container finished" podID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" containerID="9429d0f2363718a2d9dcd424fdc1cf29106d81c957f0666016e9fecdc43df02d" exitCode=0 Jul 10 21:31:38.185551 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.185520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b465d987-cs7mt" event={"ID":"aa1ea812-9aa8-4546-be22-d27ee389b3c2","Type":"ContainerDied","Data":"9429d0f2363718a2d9dcd424fdc1cf29106d81c957f0666016e9fecdc43df02d"} Jul 10 21:31:38.286556 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.286491 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:31:38.382526 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.382493 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm59w\" (UniqueName: \"kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w\") pod \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " Jul 10 21:31:38.382705 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.382555 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert\") pod \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\" (UID: \"aa1ea812-9aa8-4546-be22-d27ee389b3c2\") " Jul 10 21:31:38.384823 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.384784 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w" (OuterVolumeSpecName: "kube-api-access-xm59w") pod "aa1ea812-9aa8-4546-be22-d27ee389b3c2" (UID: "aa1ea812-9aa8-4546-be22-d27ee389b3c2"). InnerVolumeSpecName "kube-api-access-xm59w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:31:38.393597 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.393566 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "aa1ea812-9aa8-4546-be22-d27ee389b3c2" (UID: "aa1ea812-9aa8-4546-be22-d27ee389b3c2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:31:38.483547 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.483504 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm59w\" (UniqueName: \"kubernetes.io/projected/aa1ea812-9aa8-4546-be22-d27ee389b3c2-kube-api-access-xm59w\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:31:38.483547 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:38.483538 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa1ea812-9aa8-4546-be22-d27ee389b3c2-tls-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:31:39.188281 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:39.188251 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b465d987-cs7mt" Jul 10 21:31:39.188281 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:39.188277 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b465d987-cs7mt" event={"ID":"aa1ea812-9aa8-4546-be22-d27ee389b3c2","Type":"ContainerDied","Data":"1f93f5f3d8d96f3e6f70d52d9b4c76a11a74ca824251717263c2b054f540b4f0"} Jul 10 21:31:39.188746 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:39.188322 2561 scope.go:117] "RemoveContainer" containerID="9429d0f2363718a2d9dcd424fdc1cf29106d81c957f0666016e9fecdc43df02d" Jul 10 21:31:39.213295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:39.213260 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:31:39.217635 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:39.217604 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54b465d987-cs7mt"] Jul 10 21:31:40.184130 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:31:40.184090 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" path="/var/lib/kubelet/pods/aa1ea812-9aa8-4546-be22-d27ee389b3c2/volumes" Jul 10 21:36:02.112219 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:36:02.112187 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:36:02.112675 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:36:02.112570 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:36:02.122386 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:36:02.122360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:36:02.122386 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:36:02.122384 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:41:02.140441 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:41:02.140367 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:41:02.142217 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:41:02.142193 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:41:02.147418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:41:02.147391 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:41:02.149121 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:41:02.149101 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:45:00.142874 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.142784 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl"] Jul 10 21:45:00.145363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.143334 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" containerName="authorino" Jul 10 21:45:00.145363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.143351 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" containerName="authorino" Jul 10 21:45:00.145363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.143410 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa1ea812-9aa8-4546-be22-d27ee389b3c2" containerName="authorino" Jul 10 21:45:00.146271 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.146252 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" Jul 10 21:45:00.148488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.148469 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:45:00.155950 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.155925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" Jul 10 21:45:00.161217 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.161191 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl"] Jul 10 21:45:00.302716 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.302662 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl"] Jul 10 21:45:00.305340 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:45:00.305293 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1223dc10_01df_402d_88fe_a09cb530b790.slice/crio-db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09 WatchSource:0}: Error finding container db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09: Status 404 returned error can't find the container with id db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09 Jul 10 21:45:00.307519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.307500 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:45:00.813254 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:00.813218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" event={"ID":"1223dc10-01df-402d-88fe-a09cb530b790","Type":"ContainerStarted","Data":"db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09"} Jul 10 21:45:02.825259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:02.825167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" event={"ID":"1223dc10-01df-402d-88fe-a09cb530b790","Type":"ContainerStarted","Data":"dc1a9be7705973f4fa0ee980803f1bd8dede20356daf8468271d9d0e6b6bd70e"} Jul 10 21:45:02.840272 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:02.840206 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" podStartSLOduration=0.682241471 podStartE2EDuration="2.840186425s" totalImagesPullingTime="2.157944954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:45:02.839318239 +0000 UTC m=+1441.189841845" watchObservedRunningTime="2026-07-10 21:45:02.840186425 +0000 UTC m=+1441.190710004" Jul 10 21:45:03.830671 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:03.830574 2561 generic.go:356] "Generic (PLEG): container finished" podID="1223dc10-01df-402d-88fe-a09cb530b790" containerID="dc1a9be7705973f4fa0ee980803f1bd8dede20356daf8468271d9d0e6b6bd70e" exitCode=0 Jul 10 21:45:03.830671 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:03.830619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" event={"ID":"1223dc10-01df-402d-88fe-a09cb530b790","Type":"ContainerDied","Data":"dc1a9be7705973f4fa0ee980803f1bd8dede20356daf8468271d9d0e6b6bd70e"} Jul 10 21:45:04.972343 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:04.972310 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" Jul 10 21:45:05.840988 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:05.840942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" event={"ID":"1223dc10-01df-402d-88fe-a09cb530b790","Type":"ContainerDied","Data":"db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09"} Jul 10 21:45:05.840988 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:05.840979 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728665-crkrl" Jul 10 21:45:05.841240 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:05.841030 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db84b1ae28dab37b419bb0cca4262b5ab15b11abcb2b0e03be6e119972e08b09" Jul 10 21:45:09.588723 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.588688 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:09.589135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.589044 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1223dc10-01df-402d-88fe-a09cb530b790" containerName="cleanup" Jul 10 21:45:09.589135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.589055 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1223dc10-01df-402d-88fe-a09cb530b790" containerName="cleanup" Jul 10 21:45:09.589225 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.589142 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1223dc10-01df-402d-88fe-a09cb530b790" containerName="cleanup" Jul 10 21:45:09.592179 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.592154 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.594902 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.594880 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-22c1dfe4-openshift-default-dockercfg-ds8h9\"" Jul 10 21:45:09.619706 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.619672 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:09.622691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622836 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622836 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622724 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622836 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622817 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622857 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622901 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.622951 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.622919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.623047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.623016 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.623105 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.623083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.723855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723813 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.723855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723863 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723884 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.723968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724124 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724548 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724484 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724707 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724647 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724706 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.724824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.724731 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.725099 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.725078 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.726504 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.726479 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.726852 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.726834 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.734843 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.734813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.735501 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.735467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token\") pod \"e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:09.903017 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:09.902973 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:10.051777 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.051748 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:10.054085 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:45:10.054024 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4c7f00_9bfc_4545_a888_56ff4afcba69.slice/crio-4aba0b7d6c0d94cfc0659025abdb3f3cd224993cf422b532e83f810f2fceeabb WatchSource:0}: Error finding container 4aba0b7d6c0d94cfc0659025abdb3f3cd224993cf422b532e83f810f2fceeabb: Status 404 returned error can't find the container with id 4aba0b7d6c0d94cfc0659025abdb3f3cd224993cf422b532e83f810f2fceeabb Jul 10 21:45:10.056303 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.056268 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:10.056368 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.056346 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:10.056408 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.056377 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:10.865137 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.865101 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerStarted","Data":"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c"} Jul 10 21:45:10.865523 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.865156 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerStarted","Data":"4aba0b7d6c0d94cfc0659025abdb3f3cd224993cf422b532e83f810f2fceeabb"} Jul 10 21:45:10.894002 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.893627 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podStartSLOduration=1.8930370380000001 podStartE2EDuration="1.893037038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:45:10.888899326 +0000 UTC m=+1449.239422905" watchObservedRunningTime="2026-07-10 21:45:10.893037038 +0000 UTC m=+1449.243560618" Jul 10 21:45:10.903185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:10.903144 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:11.908994 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:11.908964 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:12.875910 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:12.875878 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:12.876971 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:12.876949 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:21.919736 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.919701 2561 generic.go:356] "Generic (PLEG): container finished" podID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerID="5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c" exitCode=0 Jul 10 21:45:21.919736 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.919742 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerDied","Data":"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c"} Jul 10 21:45:21.920274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.920133 2561 scope.go:117] "RemoveContainer" containerID="5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c" Jul 10 21:45:21.921143 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.921106 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:21.921248 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.921181 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:21.921248 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:21.921221 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:45:22.925798 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:22.925763 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerStarted","Data":"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33"} Jul 10 21:45:23.903568 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:23.903518 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:23.904764 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:23.904734 2561 patch_prober.go:28] interesting pod/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 21:45:23.904866 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:23.904817 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 21:45:24.904341 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:24.904304 2561 patch_prober.go:28] interesting pod/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 21:45:24.904778 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:24.904379 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 21:45:25.903600 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:25.903558 2561 patch_prober.go:28] interesting pod/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 21:45:25.903789 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:25.903624 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 21:45:26.903957 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:26.903909 2561 patch_prober.go:28] interesting pod/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 21:45:26.904544 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:26.903967 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 21:45:27.876562 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:27.876526 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:27.907272 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:27.907241 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:27.952656 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:27.952627 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:41.015941 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:41.015897 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:41.016759 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:41.016190 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" containerID="cri-o://eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33" gracePeriod=30 Jul 10 21:45:42.954354 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:42.954303 2561 patch_prober.go:28] interesting pod/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 21:45:42.954809 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:42.954394 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 21:45:46.281877 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.281810 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:46.472439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472390 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472440 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472669 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472471 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472669 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472514 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472669 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472538 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472669 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472565 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472669 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472603 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472922 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472746 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.472922 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472758 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:45:46.472922 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472778 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:45:46.472922 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.472852 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket\") pod \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\" (UID: \"9d4c7f00-9bfc-4545-a888-56ff4afcba69\") " Jul 10 21:45:46.473168 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:45:46.473224 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473201 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.473312 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473275 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.473312 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473303 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.473312 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473270 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:45:46.473833 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.473796 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:45:46.474982 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.474955 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:45:46.475097 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.474979 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj" (OuterVolumeSpecName: "kube-api-access-cbjhj") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "kube-api-access-cbjhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:45:46.475372 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.475352 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:45:46.475477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.475454 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d4c7f00-9bfc-4545-a888-56ff4afcba69" (UID: "9d4c7f00-9bfc-4545-a888-56ff4afcba69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574643 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574680 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574690 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/9d4c7f00-9bfc-4545-a888-56ff4afcba69-kube-api-access-cbjhj\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574700 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574709 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:46.574741 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:46.574716 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d4c7f00-9bfc-4545-a888-56ff4afcba69-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:45:47.041188 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.041146 2561 generic.go:356] "Generic (PLEG): container finished" podID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerID="eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33" exitCode=0 Jul 10 21:45:47.041363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.041205 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" Jul 10 21:45:47.041363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.041211 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerDied","Data":"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33"} Jul 10 21:45:47.041363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.041266 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk" event={"ID":"9d4c7f00-9bfc-4545-a888-56ff4afcba69","Type":"ContainerDied","Data":"4aba0b7d6c0d94cfc0659025abdb3f3cd224993cf422b532e83f810f2fceeabb"} Jul 10 21:45:47.041363 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.041300 2561 scope.go:117] "RemoveContainer" containerID="eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33" Jul 10 21:45:47.051247 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.051224 2561 scope.go:117] "RemoveContainer" containerID="5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c" Jul 10 21:45:47.059520 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.059487 2561 scope.go:117] "RemoveContainer" containerID="eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33" Jul 10 21:45:47.059804 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:45:47.059780 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33\": container with ID starting with eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33 not found: ID does not exist" containerID="eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33" Jul 10 21:45:47.059848 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.059815 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33"} err="failed to get container status \"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33\": rpc error: code = NotFound desc = could not find container \"eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33\": container with ID starting with eedd73c6d66b52138404d643ecce85cb47dc71f2e11b3a3f43937d6c73c0bf33 not found: ID does not exist" Jul 10 21:45:47.059848 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.059833 2561 scope.go:117] "RemoveContainer" containerID="5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c" Jul 10 21:45:47.060044 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:45:47.060029 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c\": container with ID starting with 5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c not found: ID does not exist" containerID="5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c" Jul 10 21:45:47.060138 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.060048 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c"} err="failed to get container status \"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c\": rpc error: code = NotFound desc = could not find container \"5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c\": container with ID starting with 5de1720d725ee852cc5faca295bf53a0ff6168862ff765c8f5b0c8435a45631c not found: ID does not exist" Jul 10 21:45:47.063397 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.063368 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:47.067992 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.067962 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-22c1dfe4-openshift-default-7cd74c565c-j2bhk"] Jul 10 21:45:47.553013 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.552973 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:45:47.553403 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.553349 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:45:47.553403 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.553360 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:45:47.553473 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.553431 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:45:47.553473 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.553438 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:45:47.557993 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.557972 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:45:47.560305 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.560284 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:45:47.566019 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.565978 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:45:47.685881 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.685785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-65lfx\" (UID: \"cf137046-2a04-411d-9635-7d228c6a4ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:45:47.786740 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.786697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-65lfx\" (UID: \"cf137046-2a04-411d-9635-7d228c6a4ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:45:47.787387 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.787364 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-65lfx\" (UID: \"cf137046-2a04-411d-9635-7d228c6a4ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:45:47.870047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:47.869994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:45:48.015110 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:48.015055 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:45:48.015511 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:45:48.015481 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf137046_2a04_411d_9635_7d228c6a4ecb.slice/crio-ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520 WatchSource:0}: Error finding container ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520: Status 404 returned error can't find the container with id ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520 Jul 10 21:45:48.048394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:48.048360 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" event={"ID":"cf137046-2a04-411d-9635-7d228c6a4ecb","Type":"ContainerStarted","Data":"ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520"} Jul 10 21:45:48.183959 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:48.183922 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" path="/var/lib/kubelet/pods/9d4c7f00-9bfc-4545-a888-56ff4afcba69/volumes" Jul 10 21:45:49.057938 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:49.057830 2561 generic.go:356] "Generic (PLEG): container finished" podID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerID="74a7e51d8debf40be57f0dde8a4d52348efe78e58783ff41507e94099530750e" exitCode=7 Jul 10 21:45:49.057938 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:49.057896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" event={"ID":"cf137046-2a04-411d-9635-7d228c6a4ecb","Type":"ContainerDied","Data":"74a7e51d8debf40be57f0dde8a4d52348efe78e58783ff41507e94099530750e"} Jul 10 21:45:49.058385 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:49.058193 2561 scope.go:117] "RemoveContainer" containerID="74a7e51d8debf40be57f0dde8a4d52348efe78e58783ff41507e94099530750e" Jul 10 21:45:50.064470 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:50.064383 2561 generic.go:356] "Generic (PLEG): container finished" podID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerID="412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b" exitCode=7 Jul 10 21:45:50.064855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:50.064495 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" event={"ID":"cf137046-2a04-411d-9635-7d228c6a4ecb","Type":"ContainerDied","Data":"412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b"} Jul 10 21:45:50.064855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:50.064550 2561 scope.go:117] "RemoveContainer" containerID="74a7e51d8debf40be57f0dde8a4d52348efe78e58783ff41507e94099530750e" Jul 10 21:45:50.064855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:50.064732 2561 scope.go:117] "RemoveContainer" containerID="412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b" Jul 10 21:45:50.065015 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:45:50.064993 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e5f00709-65lfx_odh-ai-gateway-infra(cf137046-2a04-411d-9635-7d228c6a4ecb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" Jul 10 21:45:51.071817 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:45:51.071789 2561 scope.go:117] "RemoveContainer" containerID="412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b" Jul 10 21:45:51.072185 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:45:51.072005 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e5f00709-65lfx_odh-ai-gateway-infra(cf137046-2a04-411d-9635-7d228c6a4ecb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" Jul 10 21:46:02.166135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:02.166099 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:46:02.168713 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:02.168688 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:46:02.173536 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:02.173513 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:46:02.175981 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:02.175963 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:46:04.178033 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:04.178004 2561 scope.go:117] "RemoveContainer" containerID="412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b" Jul 10 21:46:05.138911 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:05.138878 2561 generic.go:356] "Generic (PLEG): container finished" podID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerID="44c6630817ec152ccc62f665eec8089efa83d3cbc6aab8b8a947861f9d85748f" exitCode=0 Jul 10 21:46:05.139135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:05.138951 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" event={"ID":"cf137046-2a04-411d-9635-7d228c6a4ecb","Type":"ContainerDied","Data":"44c6630817ec152ccc62f665eec8089efa83d3cbc6aab8b8a947861f9d85748f"} Jul 10 21:46:05.139135 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:05.139011 2561 scope.go:117] "RemoveContainer" containerID="412687fd7ff387adea353603b69623b3d3c99548de8275381f50f67632af7a1b" Jul 10 21:46:06.171963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:06.171927 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:46:06.286518 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:06.286489 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:46:06.343956 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:06.343909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca\") pod \"cf137046-2a04-411d-9635-7d228c6a4ecb\" (UID: \"cf137046-2a04-411d-9635-7d228c6a4ecb\") " Jul 10 21:46:06.344336 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:06.344309 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf137046-2a04-411d-9635-7d228c6a4ecb" (UID: "cf137046-2a04-411d-9635-7d228c6a4ecb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:46:06.445252 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:06.445156 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf137046-2a04-411d-9635-7d228c6a4ecb-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:46:07.153475 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:07.153443 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" Jul 10 21:46:07.153475 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:07.153471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx" event={"ID":"cf137046-2a04-411d-9635-7d228c6a4ecb","Type":"ContainerDied","Data":"ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520"} Jul 10 21:46:07.153722 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:07.153510 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce14e73adc5717305b12aeb7ad157cd9436aa5f3e3a4ff9f1f9b4a44652bf520" Jul 10 21:46:07.186755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:07.186713 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:46:07.192581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:07.192546 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-65lfx"] Jul 10 21:46:08.184318 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:08.184287 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" path="/var/lib/kubelet/pods/cf137046-2a04-411d-9635-7d228c6a4ecb/volumes" Jul 10 21:46:37.603420 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603332 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj"] Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603704 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603715 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4c7f00-9bfc-4545-a888-56ff4afcba69" containerName="istio-proxy" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603732 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603738 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603745 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603751 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603802 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.603914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.603809 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:46:37.606736 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.606715 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:37.609163 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.609139 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:46:37.614829 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.614606 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj"] Jul 10 21:46:37.718745 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.718702 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj\" (UID: \"abde8d2b-ca6b-42f8-8767-c808e7564c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:37.819206 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.819163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj\" (UID: \"abde8d2b-ca6b-42f8-8767-c808e7564c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:37.819874 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.819852 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj\" (UID: \"abde8d2b-ca6b-42f8-8767-c808e7564c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:37.919276 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:37.919221 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:38.058504 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:38.058480 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj"] Jul 10 21:46:38.060628 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:46:38.060590 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabde8d2b_ca6b_42f8_8767_c808e7564c69.slice/crio-718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1 WatchSource:0}: Error finding container 718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1: Status 404 returned error can't find the container with id 718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1 Jul 10 21:46:38.296304 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:38.296264 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" event={"ID":"abde8d2b-ca6b-42f8-8767-c808e7564c69","Type":"ContainerStarted","Data":"718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1"} Jul 10 21:46:39.302635 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:39.302597 2561 generic.go:356] "Generic (PLEG): container finished" podID="abde8d2b-ca6b-42f8-8767-c808e7564c69" containerID="55ee6317a35f1e6e1b9fe452705e755f19c6b871a75c05b205e7f3a9d5c0dbd9" exitCode=0 Jul 10 21:46:39.303042 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:39.302685 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" event={"ID":"abde8d2b-ca6b-42f8-8767-c808e7564c69","Type":"ContainerDied","Data":"55ee6317a35f1e6e1b9fe452705e755f19c6b871a75c05b205e7f3a9d5c0dbd9"} Jul 10 21:46:40.443418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:40.443390 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:40.541931 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:40.541890 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca\") pod \"abde8d2b-ca6b-42f8-8767-c808e7564c69\" (UID: \"abde8d2b-ca6b-42f8-8767-c808e7564c69\") " Jul 10 21:46:40.542319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:40.542297 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abde8d2b-ca6b-42f8-8767-c808e7564c69" (UID: "abde8d2b-ca6b-42f8-8767-c808e7564c69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:46:40.643550 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:40.643509 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abde8d2b-ca6b-42f8-8767-c808e7564c69-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:46:41.313418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:41.313383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" event={"ID":"abde8d2b-ca6b-42f8-8767-c808e7564c69","Type":"ContainerDied","Data":"718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1"} Jul 10 21:46:41.313418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:41.313417 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj" Jul 10 21:46:41.313617 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:46:41.313437 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718ca725eda2dbe0b6378bf032cd785bd835c298357403d2758569d427dfb2e1" Jul 10 21:47:03.984596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.984558 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:47:03.985131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985119 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abde8d2b-ca6b-42f8-8767-c808e7564c69" containerName="revoke-keys" Jul 10 21:47:03.985216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985136 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abde8d2b-ca6b-42f8-8767-c808e7564c69" containerName="revoke-keys" Jul 10 21:47:03.985216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985181 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:47:03.985216 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985190 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:47:03.985375 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985279 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf137046-2a04-411d-9635-7d228c6a4ecb" containerName="revoke-keys" Jul 10 21:47:03.985375 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.985291 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="abde8d2b-ca6b-42f8-8767-c808e7564c69" containerName="revoke-keys" Jul 10 21:47:03.988979 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.988956 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:03.991920 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:03.991896 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2fe53472-openshift-default-dockercfg-5l2j5\"" Jul 10 21:47:04.017794 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.017755 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:47:04.049888 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.049856 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050088 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.049894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050088 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.049927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050088 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.049974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050088 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.049990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzg7c\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050088 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.050009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050262 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.050086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050262 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.050148 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.050262 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.050180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151538 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151485 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151911 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151729 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151911 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151911 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzg7c\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.151911 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.152266 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.151938 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.152266 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.152004 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.152266 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.152111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.152266 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.152247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.152555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.152531 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.154357 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.154326 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.154549 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.154531 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.161029 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.160994 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzg7c\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.161417 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.161392 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token\") pod \"e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.299486 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.299384 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:04.448399 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.448365 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:47:04.450419 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:47:04.450391 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1820872f_010b_479b_bafc_50c7339c2320.slice/crio-89f11caefd832b379f66b9e4add4f1961cc6857f3fd5b731052a948a23a27953 WatchSource:0}: Error finding container 89f11caefd832b379f66b9e4add4f1961cc6857f3fd5b731052a948a23a27953: Status 404 returned error can't find the container with id 89f11caefd832b379f66b9e4add4f1961cc6857f3fd5b731052a948a23a27953 Jul 10 21:47:04.452526 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.452487 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:04.452639 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.452574 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:04.452639 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:04.452621 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:05.429750 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:05.429716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerStarted","Data":"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21"} Jul 10 21:47:05.430170 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:05.429772 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerStarted","Data":"89f11caefd832b379f66b9e4add4f1961cc6857f3fd5b731052a948a23a27953"} Jul 10 21:47:05.455058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:05.454996 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podStartSLOduration=2.454976224 podStartE2EDuration="2.454976224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:47:05.452927198 +0000 UTC m=+1563.803450797" watchObservedRunningTime="2026-07-10 21:47:05.454976224 +0000 UTC m=+1563.805499877" Jul 10 21:47:06.300246 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:06.300184 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:06.305100 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:06.305053 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:06.435177 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:06.435147 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:06.436166 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:06.436142 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:09.514225 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.514191 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:09.517632 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.517613 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:09.519792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.519774 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:47:09.524813 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.524785 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:09.603269 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.603222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk\" (UID: \"c4d59323-646e-4ecb-88dc-07cbd55131a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:09.704143 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.704087 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk\" (UID: \"c4d59323-646e-4ecb-88dc-07cbd55131a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:09.704742 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.704713 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk\" (UID: \"c4d59323-646e-4ecb-88dc-07cbd55131a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:09.830689 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:09.830595 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:10.020460 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:10.020376 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:10.455109 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:10.454995 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerStarted","Data":"7b780ec50b98c8bc46b1e72bc342e7fbcc7f94a95193d6bc963ee06e9a7e3ea7"} Jul 10 21:47:10.455109 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:10.455089 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerStarted","Data":"fc5f15bd76bdf62b8c79f64e711d092246ae3080206b85bff0de3982463b1914"} Jul 10 21:47:11.464855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:11.463706 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerID="7b780ec50b98c8bc46b1e72bc342e7fbcc7f94a95193d6bc963ee06e9a7e3ea7" exitCode=7 Jul 10 21:47:11.464855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:11.463879 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerDied","Data":"7b780ec50b98c8bc46b1e72bc342e7fbcc7f94a95193d6bc963ee06e9a7e3ea7"} Jul 10 21:47:11.464855 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:11.464252 2561 scope.go:117] "RemoveContainer" containerID="7b780ec50b98c8bc46b1e72bc342e7fbcc7f94a95193d6bc963ee06e9a7e3ea7" Jul 10 21:47:12.479259 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:12.474684 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerStarted","Data":"03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db"} Jul 10 21:47:12.493842 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:12.493782 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" podStartSLOduration=3.493765709 podStartE2EDuration="3.493765709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:47:10.478784616 +0000 UTC m=+1568.829308211" watchObservedRunningTime="2026-07-10 21:47:12.493765709 +0000 UTC m=+1570.844289286" Jul 10 21:47:13.481889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:13.481786 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerID="03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db" exitCode=7 Jul 10 21:47:13.481889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:13.481869 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerDied","Data":"03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db"} Jul 10 21:47:13.482409 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:13.481935 2561 scope.go:117] "RemoveContainer" containerID="7b780ec50b98c8bc46b1e72bc342e7fbcc7f94a95193d6bc963ee06e9a7e3ea7" Jul 10 21:47:13.482409 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:13.482257 2561 scope.go:117] "RemoveContainer" containerID="03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db" Jul 10 21:47:13.482503 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:47:13.482485 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk_odh-ai-gateway-infra(c4d59323-646e-4ecb-88dc-07cbd55131a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" Jul 10 21:47:14.489326 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:14.489295 2561 scope.go:117] "RemoveContainer" containerID="03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db" Jul 10 21:47:14.489708 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:47:14.489492 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk_odh-ai-gateway-infra(c4d59323-646e-4ecb-88dc-07cbd55131a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" Jul 10 21:47:16.091409 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:47:16.091352 2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1820872f_010b_479b_bafc_50c7339c2320.slice/crio-conmon-792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21.scope\": RecentStats: unable to find data in memory cache]" Jul 10 21:47:16.500333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.500294 2561 generic.go:356] "Generic (PLEG): container finished" podID="1820872f-010b-479b-bafc-50c7339c2320" containerID="792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21" exitCode=0 Jul 10 21:47:16.500524 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.500365 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerDied","Data":"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21"} Jul 10 21:47:16.500808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.500791 2561 scope.go:117] "RemoveContainer" containerID="792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21" Jul 10 21:47:16.501916 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.501887 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:16.501981 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.501956 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:16.502018 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:16.501987 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:47:17.506834 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:17.506799 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerStarted","Data":"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f"} Jul 10 21:47:18.299747 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:18.299714 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:18.301089 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:18.301043 2561 patch_prober.go:28] interesting pod/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 10 21:47:18.301203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:18.301114 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 10 21:47:19.300273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:19.300231 2561 patch_prober.go:28] interesting pod/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 10 21:47:19.300777 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:19.300302 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 10 21:47:20.300087 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:20.300020 2561 patch_prober.go:28] interesting pod/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 10 21:47:20.300280 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:20.300112 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 10 21:47:21.300056 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:21.300008 2561 patch_prober.go:28] interesting pod/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 10 21:47:21.300498 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:21.300110 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 10 21:47:21.435927 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:21.435885 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:22.304022 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:22.303989 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:22.304991 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:22.304969 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:47:27.178366 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:27.178324 2561 scope.go:117] "RemoveContainer" containerID="03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db" Jul 10 21:47:27.554965 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:27.554927 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerStarted","Data":"0e76fe142ec5ee9bd10d09636103abea85b8ab0cf1bdc3a36a728da6666a5deb"} Jul 10 21:47:28.562258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:28.562221 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerID="0e76fe142ec5ee9bd10d09636103abea85b8ab0cf1bdc3a36a728da6666a5deb" exitCode=0 Jul 10 21:47:28.562641 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:28.562294 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerDied","Data":"0e76fe142ec5ee9bd10d09636103abea85b8ab0cf1bdc3a36a728da6666a5deb"} Jul 10 21:47:28.562641 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:28.562382 2561 scope.go:117] "RemoveContainer" containerID="03b4cca67207551a1c73cb1f1d5441063de0b04d02a61c3501eb3d7501fe86db" Jul 10 21:47:28.580028 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:28.579992 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:29.713862 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:29.713837 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:29.796207 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:29.796169 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca\") pod \"c4d59323-646e-4ecb-88dc-07cbd55131a4\" (UID: \"c4d59323-646e-4ecb-88dc-07cbd55131a4\") " Jul 10 21:47:29.796555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:29.796532 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4d59323-646e-4ecb-88dc-07cbd55131a4" (UID: "c4d59323-646e-4ecb-88dc-07cbd55131a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:47:29.897652 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:29.897618 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d59323-646e-4ecb-88dc-07cbd55131a4-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:47:30.577476 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:30.577390 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" Jul 10 21:47:30.577617 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:30.577391 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk" event={"ID":"c4d59323-646e-4ecb-88dc-07cbd55131a4","Type":"ContainerDied","Data":"fc5f15bd76bdf62b8c79f64e711d092246ae3080206b85bff0de3982463b1914"} Jul 10 21:47:30.577617 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:30.577560 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5f15bd76bdf62b8c79f64e711d092246ae3080206b85bff0de3982463b1914" Jul 10 21:47:30.608484 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:30.608440 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:30.610114 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:30.610082 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-tmsnk"] Jul 10 21:47:32.184454 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:47:32.184408 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" path="/var/lib/kubelet/pods/c4d59323-646e-4ecb-88dc-07cbd55131a4/volumes" Jul 10 21:48:09.558392 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558350 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz"] Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558711 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558722 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558731 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558737 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558744 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558750 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558818 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.558978 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.558825 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:48:09.560953 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.560936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:09.563258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.563236 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:48:09.569871 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.569838 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz"] Jul 10 21:48:09.659370 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.659332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-w92pz\" (UID: \"d7958275-9f52-45a3-8a9b-8674e430aef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:09.760868 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.760823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-w92pz\" (UID: \"d7958275-9f52-45a3-8a9b-8674e430aef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:09.761473 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.761453 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2fe53472-w92pz\" (UID: \"d7958275-9f52-45a3-8a9b-8674e430aef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:09.871691 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:09.871656 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:10.014618 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:10.014545 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz"] Jul 10 21:48:10.016890 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:48:10.016860 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7958275_9f52_45a3_8a9b_8674e430aef6.slice/crio-9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60 WatchSource:0}: Error finding container 9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60: Status 404 returned error can't find the container with id 9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60 Jul 10 21:48:10.773718 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:10.773681 2561 generic.go:356] "Generic (PLEG): container finished" podID="d7958275-9f52-45a3-8a9b-8674e430aef6" containerID="65f35b65bf52a8c9effd205649e3f58f552206f72fe0b8b45311e250297302ab" exitCode=0 Jul 10 21:48:10.774157 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:10.773773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" event={"ID":"d7958275-9f52-45a3-8a9b-8674e430aef6","Type":"ContainerDied","Data":"65f35b65bf52a8c9effd205649e3f58f552206f72fe0b8b45311e250297302ab"} Jul 10 21:48:10.774157 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:10.773841 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" event={"ID":"d7958275-9f52-45a3-8a9b-8674e430aef6","Type":"ContainerStarted","Data":"9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60"} Jul 10 21:48:11.910549 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:11.910519 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:11.981725 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:11.981688 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca\") pod \"d7958275-9f52-45a3-8a9b-8674e430aef6\" (UID: \"d7958275-9f52-45a3-8a9b-8674e430aef6\") " Jul 10 21:48:11.982090 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:11.982040 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7958275-9f52-45a3-8a9b-8674e430aef6" (UID: "d7958275-9f52-45a3-8a9b-8674e430aef6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:48:12.082918 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:12.082828 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7958275-9f52-45a3-8a9b-8674e430aef6-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:12.784605 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:12.784573 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" Jul 10 21:48:12.784792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:12.784573 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2fe53472-w92pz" event={"ID":"d7958275-9f52-45a3-8a9b-8674e430aef6","Type":"ContainerDied","Data":"9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60"} Jul 10 21:48:12.784792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:12.784697 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9902f06297c00b26810b5d96319fd773004b5a64e927edcb08482dfc570d7e60" Jul 10 21:48:34.804383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:34.804345 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:48:34.804860 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:34.804585 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" containerID="cri-o://a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f" gracePeriod=30 Jul 10 21:48:37.306865 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:37.306819 2561 patch_prober.go:28] interesting pod/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 21:48:37.307258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:37.306894 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 21:48:40.069188 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.069118 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:48:40.132914 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.132876 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.132935 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.132972 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.132997 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133040 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133108 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzg7c\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133136 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133159 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133188 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data\") pod \"1820872f-010b-479b-bafc-50c7339c2320\" (UID: \"1820872f-010b-479b-bafc-50c7339c2320\") " Jul 10 21:48:40.133391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133338 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:48:40.133391 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133357 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:48:40.133640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133468 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:48:40.133640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133564 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.133640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133583 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.133957 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.133668 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:48:40.134383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.134356 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data" (OuterVolumeSpecName: "istio-data") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:48:40.135382 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.135352 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:48:40.135484 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.135441 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:48:40.135484 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.135444 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token" (OuterVolumeSpecName: "istio-token") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:48:40.135769 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.135746 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c" (OuterVolumeSpecName: "kube-api-access-bzg7c") pod "1820872f-010b-479b-bafc-50c7339c2320" (UID: "1820872f-010b-479b-bafc-50c7339c2320"). InnerVolumeSpecName "kube-api-access-bzg7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:48:40.235058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235012 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235048 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1820872f-010b-479b-bafc-50c7339c2320-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235058 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1820872f-010b-479b-bafc-50c7339c2320-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235093 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzg7c\" (UniqueName: \"kubernetes.io/projected/1820872f-010b-479b-bafc-50c7339c2320-kube-api-access-bzg7c\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235412 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235104 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235412 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235113 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.235412 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.235121 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1820872f-010b-479b-bafc-50c7339c2320-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:48:40.924597 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.924560 2561 generic.go:356] "Generic (PLEG): container finished" podID="1820872f-010b-479b-bafc-50c7339c2320" containerID="a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f" exitCode=0 Jul 10 21:48:40.924765 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.924616 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" Jul 10 21:48:40.924765 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.924646 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerDied","Data":"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f"} Jul 10 21:48:40.924765 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.924698 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k" event={"ID":"1820872f-010b-479b-bafc-50c7339c2320","Type":"ContainerDied","Data":"89f11caefd832b379f66b9e4add4f1961cc6857f3fd5b731052a948a23a27953"} Jul 10 21:48:40.924765 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.924720 2561 scope.go:117] "RemoveContainer" containerID="a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f" Jul 10 21:48:40.933651 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.933628 2561 scope.go:117] "RemoveContainer" containerID="792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21" Jul 10 21:48:40.942383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.942353 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:48:40.942986 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.942969 2561 scope.go:117] "RemoveContainer" containerID="a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f" Jul 10 21:48:40.943465 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:48:40.943377 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f\": container with ID starting with a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f not found: ID does not exist" containerID="a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f" Jul 10 21:48:40.943593 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.943524 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f"} err="failed to get container status \"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f\": rpc error: code = NotFound desc = could not find container \"a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f\": container with ID starting with a213cae29a00ba3d9a0b502f543e1ae8cc85a7090122cd139a0432c2914f187f not found: ID does not exist" Jul 10 21:48:40.943667 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.943595 2561 scope.go:117] "RemoveContainer" containerID="792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21" Jul 10 21:48:40.944038 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:48:40.943977 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21\": container with ID starting with 792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21 not found: ID does not exist" containerID="792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21" Jul 10 21:48:40.944038 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.944012 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21"} err="failed to get container status \"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21\": rpc error: code = NotFound desc = could not find container \"792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21\": container with ID starting with 792be6b8c715a7bcdbc6036620fabffc817bd1b222d4c7df370d1bd5ae8dde21 not found: ID does not exist" Jul 10 21:48:40.945960 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:40.945937 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2fe53472-openshift-default-d578969c5-pmz2k"] Jul 10 21:48:42.184319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:48:42.184278 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1820872f-010b-479b-bafc-50c7339c2320" path="/var/lib/kubelet/pods/1820872f-010b-479b-bafc-50c7339c2320/volumes" Jul 10 21:50:10.165686 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.165645 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:10.166320 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166191 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:10.166320 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166213 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:10.166320 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166252 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7958275-9f52-45a3-8a9b-8674e430aef6" containerName="revoke-keys" Jul 10 21:50:10.166320 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166261 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7958275-9f52-45a3-8a9b-8674e430aef6" containerName="revoke-keys" Jul 10 21:50:10.166581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166337 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:10.166581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166351 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:10.166581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166360 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7958275-9f52-45a3-8a9b-8674e430aef6" containerName="revoke-keys" Jul 10 21:50:10.166581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.166371 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d59323-646e-4ecb-88dc-07cbd55131a4" containerName="revoke-keys" Jul 10 21:50:10.169435 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.169414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.171670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.171649 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fa2f395f-openshift-default-dockercfg-bftlt\"" Jul 10 21:50:10.192542 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.192501 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:10.245130 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245087 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245224 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245270 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhcn\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245509 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245387 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245509 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.245509 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.245456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.345999 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.345961 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.345999 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.345996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhcn\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346176 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346527 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346527 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346527 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346435 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346678 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346650 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346739 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346681 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.346739 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.346705 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.347022 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.347000 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.348557 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.348536 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.348880 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.348862 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.353990 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.353955 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.354677 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.354657 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhcn\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn\") pod \"e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.479886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.479793 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:10.625714 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.625687 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:10.628425 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:50:10.628396 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46578702_0c19_4add_a9ea_11158b1983ce.slice/crio-4340ad4e43b722598c21b5d7bf469fa1eedb9f6ab2a96683fe39caef6cbb0bd0 WatchSource:0}: Error finding container 4340ad4e43b722598c21b5d7bf469fa1eedb9f6ab2a96683fe39caef6cbb0bd0: Status 404 returned error can't find the container with id 4340ad4e43b722598c21b5d7bf469fa1eedb9f6ab2a96683fe39caef6cbb0bd0 Jul 10 21:50:10.630589 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.630551 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:10.630682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.630629 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:10.630682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:10.630662 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:11.351660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:11.351618 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerStarted","Data":"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b"} Jul 10 21:50:11.352026 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:11.351685 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerStarted","Data":"4340ad4e43b722598c21b5d7bf469fa1eedb9f6ab2a96683fe39caef6cbb0bd0"} Jul 10 21:50:11.375178 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:11.375119 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podStartSLOduration=1.375099786 podStartE2EDuration="1.375099786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:50:11.372981189 +0000 UTC m=+1749.723504786" watchObservedRunningTime="2026-07-10 21:50:11.375099786 +0000 UTC m=+1749.725623363" Jul 10 21:50:11.479940 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:11.479895 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:11.490477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:11.490442 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:12.356447 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:12.356413 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:12.357472 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:12.357452 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:24.755446 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.755399 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z"] Jul 10 21:50:24.755964 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.755942 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:24.756035 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.755968 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1820872f-010b-479b-bafc-50c7339c2320" containerName="istio-proxy" Jul 10 21:50:24.759462 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.759433 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:24.762010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.761979 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:50:24.768200 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.768162 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z"] Jul 10 21:50:24.884285 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.884235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z\" (UID: \"a0b1105d-1c52-40e8-a636-c35bc2b9faef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:24.985161 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.985116 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z\" (UID: \"a0b1105d-1c52-40e8-a636-c35bc2b9faef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:24.986417 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:24.986382 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z\" (UID: \"a0b1105d-1c52-40e8-a636-c35bc2b9faef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:25.076584 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:25.076470 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:25.246640 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:25.246608 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z"] Jul 10 21:50:25.249101 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:50:25.249053 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b1105d_1c52_40e8_a636_c35bc2b9faef.slice/crio-3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b WatchSource:0}: Error finding container 3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b: Status 404 returned error can't find the container with id 3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b Jul 10 21:50:25.420510 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:25.420470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" event={"ID":"a0b1105d-1c52-40e8-a636-c35bc2b9faef","Type":"ContainerStarted","Data":"3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b"} Jul 10 21:50:26.428540 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:26.428503 2561 generic.go:356] "Generic (PLEG): container finished" podID="a0b1105d-1c52-40e8-a636-c35bc2b9faef" containerID="83456add3445aac4ec58659a9d44391d89222aaf3c49be8ff77af0af97c03629" exitCode=0 Jul 10 21:50:26.428926 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:26.428595 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" event={"ID":"a0b1105d-1c52-40e8-a636-c35bc2b9faef","Type":"ContainerDied","Data":"83456add3445aac4ec58659a9d44391d89222aaf3c49be8ff77af0af97c03629"} Jul 10 21:50:27.572828 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:27.572801 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:27.712402 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:27.712313 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca\") pod \"a0b1105d-1c52-40e8-a636-c35bc2b9faef\" (UID: \"a0b1105d-1c52-40e8-a636-c35bc2b9faef\") " Jul 10 21:50:27.712711 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:27.712689 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0b1105d-1c52-40e8-a636-c35bc2b9faef" (UID: "a0b1105d-1c52-40e8-a636-c35bc2b9faef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:50:27.813449 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:27.813407 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b1105d-1c52-40e8-a636-c35bc2b9faef-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:28.250918 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.250881 2561 patch_prober.go:28] interesting pod/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:57222->10.132.0.42:15021: read: connection reset by peer" start-of-body= Jul 10 21:50:28.251319 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.250956 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:57222->10.132.0.42:15021: read: connection reset by peer" Jul 10 21:50:28.439554 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.439471 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" Jul 10 21:50:28.439554 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.439511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z" event={"ID":"a0b1105d-1c52-40e8-a636-c35bc2b9faef","Type":"ContainerDied","Data":"3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b"} Jul 10 21:50:28.439740 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.439567 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3624de1049b5cea8ed97b9961484f7a68950a1c6d1ca05bb29b30b7d7cfe5b" Jul 10 21:50:28.441211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.441145 2561 generic.go:356] "Generic (PLEG): container finished" podID="46578702-0c19-4add-a9ea-11158b1983ce" containerID="86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b" exitCode=0 Jul 10 21:50:28.441211 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.441197 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerDied","Data":"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b"} Jul 10 21:50:28.441677 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.441655 2561 scope.go:117] "RemoveContainer" containerID="86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b" Jul 10 21:50:28.442700 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.442672 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:28.442751 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.442739 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:28.442788 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:28.442775 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:50:29.448873 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:29.448833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerStarted","Data":"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157"} Jul 10 21:50:29.486533 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:29.486496 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:29.486863 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:29.486622 2561 patch_prober.go:28] interesting pod/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 21:50:29.487030 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:29.486997 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 21:50:30.480738 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:30.480683 2561 patch_prober.go:28] interesting pod/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 21:50:30.481317 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:30.480762 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 21:50:31.480711 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:31.480665 2561 patch_prober.go:28] interesting pod/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 21:50:31.480930 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:31.480741 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 21:50:32.481085 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:32.481018 2561 patch_prober.go:28] interesting pod/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 21:50:32.481612 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:32.481124 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 21:50:33.484378 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:33.484347 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:33.484779 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:33.484734 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:33.485585 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:33.485566 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:50.481436 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:50.481397 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:50.482078 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:50.481644 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" containerID="cri-o://88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157" gracePeriod=30 Jul 10 21:50:55.746256 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.746187 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:55.771447 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771415 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771447 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771453 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771515 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771544 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771578 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771612 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771682 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771664 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vhcn\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771925 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771690 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771925 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771733 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert\") pod \"46578702-0c19-4add-a9ea-11158b1983ce\" (UID: \"46578702-0c19-4add-a9ea-11158b1983ce\") " Jul 10 21:50:55.771925 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.771875 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:50:55.772130 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.772043 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.772189 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.772129 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:50:55.772272 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.772240 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:50:55.772446 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.772423 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:50:55.772924 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.772898 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:50:55.773972 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.773948 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:50:55.774432 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.774408 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn" (OuterVolumeSpecName: "kube-api-access-4vhcn") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "kube-api-access-4vhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:50:55.774557 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.774533 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:50:55.774615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.774555 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46578702-0c19-4add-a9ea-11158b1983ce" (UID: "46578702-0c19-4add-a9ea-11158b1983ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:50:55.872609 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872563 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872609 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872601 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872609 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872615 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872627 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vhcn\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-kube-api-access-4vhcn\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872638 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46578702-0c19-4add-a9ea-11158b1983ce-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872651 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46578702-0c19-4add-a9ea-11158b1983ce-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872661 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46578702-0c19-4add-a9ea-11158b1983ce-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:55.872861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:55.872671 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46578702-0c19-4add-a9ea-11158b1983ce-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:50:56.586900 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.586817 2561 generic.go:356] "Generic (PLEG): container finished" podID="46578702-0c19-4add-a9ea-11158b1983ce" containerID="88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157" exitCode=0 Jul 10 21:50:56.586900 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.586868 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" Jul 10 21:50:56.587141 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.586903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerDied","Data":"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157"} Jul 10 21:50:56.587141 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.586959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw" event={"ID":"46578702-0c19-4add-a9ea-11158b1983ce","Type":"ContainerDied","Data":"4340ad4e43b722598c21b5d7bf469fa1eedb9f6ab2a96683fe39caef6cbb0bd0"} Jul 10 21:50:56.587141 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.586983 2561 scope.go:117] "RemoveContainer" containerID="88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157" Jul 10 21:50:56.596456 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.596434 2561 scope.go:117] "RemoveContainer" containerID="86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b" Jul 10 21:50:56.606349 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.606260 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:56.608433 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.608410 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fa2f395f-openshift-default-6d567b66c-lq7lw"] Jul 10 21:50:56.608953 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.608854 2561 scope.go:117] "RemoveContainer" containerID="88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157" Jul 10 21:50:56.609222 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:50:56.609200 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157\": container with ID starting with 88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157 not found: ID does not exist" containerID="88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157" Jul 10 21:50:56.609277 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.609231 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157"} err="failed to get container status \"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157\": rpc error: code = NotFound desc = could not find container \"88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157\": container with ID starting with 88748d5e18bca10501e583dc812d6f4fc258f9b17203fbf1a370c5732d5a3157 not found: ID does not exist" Jul 10 21:50:56.609277 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.609251 2561 scope.go:117] "RemoveContainer" containerID="86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b" Jul 10 21:50:56.609516 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:50:56.609497 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b\": container with ID starting with 86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b not found: ID does not exist" containerID="86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b" Jul 10 21:50:56.609573 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:56.609521 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b"} err="failed to get container status \"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b\": rpc error: code = NotFound desc = could not find container \"86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b\": container with ID starting with 86ed0793f95df1fd8b6abfe09ba6b0726fced9ffef7bfeb51bbb6ebf9b5fd10b not found: ID does not exist" Jul 10 21:50:58.184838 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:50:58.184797 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46578702-0c19-4add-a9ea-11158b1983ce" path="/var/lib/kubelet/pods/46578702-0c19-4add-a9ea-11158b1983ce/volumes" Jul 10 21:51:02.194894 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:02.194861 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:51:02.197273 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:02.197251 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:51:02.201937 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:02.201913 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:51:02.204268 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:02.204246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:51:09.850685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.850585 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:51:09.851131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851114 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:09.851185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851134 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:09.851185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851159 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b1105d-1c52-40e8-a636-c35bc2b9faef" containerName="revoke-keys" Jul 10 21:51:09.851185 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851167 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b1105d-1c52-40e8-a636-c35bc2b9faef" containerName="revoke-keys" Jul 10 21:51:09.851277 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851262 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0b1105d-1c52-40e8-a636-c35bc2b9faef" containerName="revoke-keys" Jul 10 21:51:09.851311 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851276 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:09.851311 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.851287 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:09.854772 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.854753 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.857361 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.857335 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-acd5952a-openshift-default-dockercfg-ztqkm\"" Jul 10 21:51:09.864309 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.864279 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:51:09.997224 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997224 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997269 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppk7\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:09.997477 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:09.997446 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.098766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.098766 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098765 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppk7\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098852 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099034 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.098937 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099377 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099377 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099341 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099463 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099380 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099463 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099444 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099649 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099625 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.099775 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.099759 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.101439 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.101384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.101625 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.101607 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.107258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.107227 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.107356 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.107319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppk7\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7\") pod \"e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.165295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.165244 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:10.314320 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.314279 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:51:10.317700 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:51:10.317668 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565e1e32_222a_493d_8a40_6a8f7a0aabaa.slice/crio-c9af8e0c1cdb8668afaea503813f3f2efc27bf18e3a121eff02472bd5309d2a1 WatchSource:0}: Error finding container c9af8e0c1cdb8668afaea503813f3f2efc27bf18e3a121eff02472bd5309d2a1: Status 404 returned error can't find the container with id c9af8e0c1cdb8668afaea503813f3f2efc27bf18e3a121eff02472bd5309d2a1 Jul 10 21:51:10.320389 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.320350 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:10.320483 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.320426 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:10.320483 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.320457 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:10.656660 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.656619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerStarted","Data":"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719"} Jul 10 21:51:10.656850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.656684 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerStarted","Data":"c9af8e0c1cdb8668afaea503813f3f2efc27bf18e3a121eff02472bd5309d2a1"} Jul 10 21:51:10.677566 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:10.677507 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podStartSLOduration=1.67748877 podStartE2EDuration="1.67748877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:51:10.675867695 +0000 UTC m=+1809.026391302" watchObservedRunningTime="2026-07-10 21:51:10.67748877 +0000 UTC m=+1809.028012348" Jul 10 21:51:11.166295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:11.166254 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:11.171825 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:11.171795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:11.661434 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:11.661392 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:11.662336 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:11.662313 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:23.722158 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.722117 2561 generic.go:356] "Generic (PLEG): container finished" podID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerID="0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719" exitCode=0 Jul 10 21:51:23.722598 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.722189 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerDied","Data":"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719"} Jul 10 21:51:23.722647 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.722611 2561 scope.go:117] "RemoveContainer" containerID="0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719" Jul 10 21:51:23.723688 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.723636 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:23.723746 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.723718 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:23.723796 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:23.723772 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:51:24.729648 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:24.729611 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerStarted","Data":"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808"} Jul 10 21:51:25.166108 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:25.166047 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:25.167745 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:25.167713 2561 patch_prober.go:28] interesting pod/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 21:51:25.167996 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:25.167966 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 21:51:26.166301 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:26.166258 2561 patch_prober.go:28] interesting pod/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 21:51:26.166929 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:26.166894 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 21:51:26.662383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:26.662338 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:27.165863 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:27.165823 2561 patch_prober.go:28] interesting pod/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 21:51:27.166200 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:27.166167 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 21:51:28.166739 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:28.166697 2561 patch_prober.go:28] interesting pod/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 21:51:28.167812 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:28.167349 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 21:51:29.169436 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:29.169404 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:29.170534 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:29.170515 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:51:55.229390 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.229350 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn"] Jul 10 21:51:55.229864 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.229703 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:55.229864 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.229713 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="46578702-0c19-4add-a9ea-11158b1983ce" containerName="istio-proxy" Jul 10 21:51:55.232775 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.232755 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:55.235131 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.235102 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:51:55.240720 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.240696 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn"] Jul 10 21:51:55.349385 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.349327 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn\" (UID: \"d7567810-9346-4f1c-8a39-b639e95ecb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:55.451019 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.450976 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn\" (UID: \"d7567810-9346-4f1c-8a39-b639e95ecb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:55.451814 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.451784 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn\" (UID: \"d7567810-9346-4f1c-8a39-b639e95ecb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:55.544636 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.544543 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:55.684899 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.684871 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn"] Jul 10 21:51:55.687481 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:51:55.687450 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7567810_9346_4f1c_8a39_b639e95ecb2a.slice/crio-f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103 WatchSource:0}: Error finding container f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103: Status 404 returned error can't find the container with id f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103 Jul 10 21:51:55.881586 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:55.881553 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" event={"ID":"d7567810-9346-4f1c-8a39-b639e95ecb2a","Type":"ContainerStarted","Data":"f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103"} Jul 10 21:51:56.888338 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:56.888304 2561 generic.go:356] "Generic (PLEG): container finished" podID="d7567810-9346-4f1c-8a39-b639e95ecb2a" containerID="b36ff27ae390cdc1b831866a35b54dd6bb191f3b4a34476853235a332a86a786" exitCode=0 Jul 10 21:51:56.888722 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:56.888395 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" event={"ID":"d7567810-9346-4f1c-8a39-b639e95ecb2a","Type":"ContainerDied","Data":"b36ff27ae390cdc1b831866a35b54dd6bb191f3b4a34476853235a332a86a786"} Jul 10 21:51:58.031584 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.031556 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:51:58.176244 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.176142 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca\") pod \"d7567810-9346-4f1c-8a39-b639e95ecb2a\" (UID: \"d7567810-9346-4f1c-8a39-b639e95ecb2a\") " Jul 10 21:51:58.176581 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.176550 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7567810-9346-4f1c-8a39-b639e95ecb2a" (UID: "d7567810-9346-4f1c-8a39-b639e95ecb2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:51:58.277560 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.277524 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7567810-9346-4f1c-8a39-b639e95ecb2a-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:51:58.899798 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.899763 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" event={"ID":"d7567810-9346-4f1c-8a39-b639e95ecb2a","Type":"ContainerDied","Data":"f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103"} Jul 10 21:51:58.899973 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.899810 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4dbc3b539316ffe65298ad18b6b8de5d17e67c0e72ca8ca7c93a3277a78a103" Jul 10 21:51:58.899973 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:51:58.899823 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn" Jul 10 21:53:02.285118 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:02.285019 2561 scope.go:117] "RemoveContainer" containerID="44c6630817ec152ccc62f665eec8089efa83d3cbc6aab8b8a947861f9d85748f" Jul 10 21:53:02.594567 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:02.594472 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:53:02.594722 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:02.594706 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" containerID="cri-o://b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808" gracePeriod=30 Jul 10 21:53:07.852013 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.851985 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:53:07.903862 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.903820 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.903884 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.903909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.903953 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dppk7\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.903998 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904045 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904028 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904099 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904140 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904181 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs\") pod \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\" (UID: \"565e1e32-222a-493d-8a40-6a8f7a0aabaa\") " Jul 10 21:53:07.904330 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904252 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:53:07.904534 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904347 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:53:07.904534 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904476 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:53:07.904644 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904581 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:07.904644 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904603 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:07.904644 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904622 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:07.904930 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904819 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:53:07.905010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.904985 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:53:07.906466 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.906438 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:53:07.906572 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.906441 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:53:07.906956 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.906933 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:53:07.907022 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:07.906937 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7" (OuterVolumeSpecName: "kube-api-access-dppk7") pod "565e1e32-222a-493d-8a40-6a8f7a0aabaa" (UID: "565e1e32-222a-493d-8a40-6a8f7a0aabaa"). InnerVolumeSpecName "kube-api-access-dppk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:53:08.005850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005809 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.005850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005842 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.005850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005851 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppk7\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-kube-api-access-dppk7\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.005850 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005863 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.006163 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005871 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.006163 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.005879 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1e32-222a-493d-8a40-6a8f7a0aabaa-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:53:08.254886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.254851 2561 generic.go:356] "Generic (PLEG): container finished" podID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerID="b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808" exitCode=0 Jul 10 21:53:08.255079 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.254906 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" Jul 10 21:53:08.255079 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.254939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerDied","Data":"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808"} Jul 10 21:53:08.255079 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.254994 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25" event={"ID":"565e1e32-222a-493d-8a40-6a8f7a0aabaa","Type":"ContainerDied","Data":"c9af8e0c1cdb8668afaea503813f3f2efc27bf18e3a121eff02472bd5309d2a1"} Jul 10 21:53:08.255079 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.255019 2561 scope.go:117] "RemoveContainer" containerID="b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808" Jul 10 21:53:08.264009 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.263985 2561 scope.go:117] "RemoveContainer" containerID="0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719" Jul 10 21:53:08.272767 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.272745 2561 scope.go:117] "RemoveContainer" containerID="b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808" Jul 10 21:53:08.273049 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:53:08.273030 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808\": container with ID starting with b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808 not found: ID does not exist" containerID="b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808" Jul 10 21:53:08.273139 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.273084 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808"} err="failed to get container status \"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808\": rpc error: code = NotFound desc = could not find container \"b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808\": container with ID starting with b69b60a552c1cd2e13e4d9af75ab34b9df90d4cd5e89891ccb5055a47c92f808 not found: ID does not exist" Jul 10 21:53:08.273139 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.273104 2561 scope.go:117] "RemoveContainer" containerID="0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719" Jul 10 21:53:08.273343 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:53:08.273327 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719\": container with ID starting with 0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719 not found: ID does not exist" containerID="0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719" Jul 10 21:53:08.273394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.273349 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719"} err="failed to get container status \"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719\": rpc error: code = NotFound desc = could not find container \"0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719\": container with ID starting with 0500a01c187acb2e3f7070c20004e48e017cda347d30970e7f5df72e91b73719 not found: ID does not exist" Jul 10 21:53:08.285494 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.285451 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:53:08.305789 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:08.305748 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-acd5952a-openshift-default-66c9cdfcdc-dqr25"] Jul 10 21:53:10.184009 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:10.183972 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" path="/var/lib/kubelet/pods/565e1e32-222a-493d-8a40-6a8f7a0aabaa/volumes" Jul 10 21:53:57.094733 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.094653 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:53:57.095238 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095170 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:53:57.095238 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095187 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:53:57.095238 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095196 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7567810-9346-4f1c-8a39-b639e95ecb2a" containerName="revoke-keys" Jul 10 21:53:57.095238 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095202 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7567810-9346-4f1c-8a39-b639e95ecb2a" containerName="revoke-keys" Jul 10 21:53:57.095384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095277 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:53:57.095384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095284 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:53:57.095384 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.095295 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7567810-9346-4f1c-8a39-b639e95ecb2a" containerName="revoke-keys" Jul 10 21:53:57.098587 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.098569 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.101075 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.101040 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0b0348-openshift-default-dockercfg-pltzn\"" Jul 10 21:53:57.109236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.109207 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:53:57.143275 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143233 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmthc\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143294 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143382 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143489 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143457 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143701 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143701 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143529 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143701 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.143701 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.143593 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244297 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244337 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244340 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244529 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244607 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244829 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmthc\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244829 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244829 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244804 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.244987 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.244887 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.245089 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.245050 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.245333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.245128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.245333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.245213 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.245333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.245352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.247049 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.247028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.247555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.247533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.252716 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.252685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.252824 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.252797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmthc\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc\") pod \"e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.409311 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.409278 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:57.585736 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.585707 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:53:57.588300 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:53:57.588273 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03af1dba_43ef_4baf_b419_b8de686544fc.slice/crio-7e0721a1543efae7ea7980dd017d504163363b43570e1bd9b418134b80a344f3 WatchSource:0}: Error finding container 7e0721a1543efae7ea7980dd017d504163363b43570e1bd9b418134b80a344f3: Status 404 returned error can't find the container with id 7e0721a1543efae7ea7980dd017d504163363b43570e1bd9b418134b80a344f3 Jul 10 21:53:57.590517 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.590481 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:53:57.590620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.590551 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:53:57.590620 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:57.590583 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:53:58.503380 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:58.503343 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerStarted","Data":"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634"} Jul 10 21:53:58.503790 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:58.503403 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerStarted","Data":"7e0721a1543efae7ea7980dd017d504163363b43570e1bd9b418134b80a344f3"} Jul 10 21:53:58.525963 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:58.525904 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podStartSLOduration=1.525888545 podStartE2EDuration="1.525888545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:53:58.524582747 +0000 UTC m=+1976.875106330" watchObservedRunningTime="2026-07-10 21:53:58.525888545 +0000 UTC m=+1976.876412188" Jul 10 21:53:59.410248 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:59.410199 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:59.415681 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:59.415649 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:59.515037 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:59.515000 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:53:59.516174 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:53:59.516154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:54:02.306853 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:02.306821 2561 scope.go:117] "RemoveContainer" containerID="0e76fe142ec5ee9bd10d09636103abea85b8ab0cf1bdc3a36a728da6666a5deb" Jul 10 21:54:15.558278 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:15.558220 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 21:54:15.558808 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:15.558313 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 21:54:16.609459 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.609423 2561 generic.go:356] "Generic (PLEG): container finished" podID="03af1dba-43ef-4baf-b419-b8de686544fc" containerID="43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634" exitCode=0 Jul 10 21:54:16.609883 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.609495 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerDied","Data":"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634"} Jul 10 21:54:16.609883 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.609874 2561 scope.go:117] "RemoveContainer" containerID="43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634" Jul 10 21:54:16.610896 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.610866 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:54:16.610988 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.610936 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:54:16.610988 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:16.610967 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:54:17.616810 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:17.616776 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerStarted","Data":"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96"} Jul 10 21:54:18.410186 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:18.410139 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:54:18.410450 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:18.410425 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 10 21:54:18.410502 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:18.410476 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 10 21:54:19.410029 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:19.409993 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 10 21:54:19.410491 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:19.410052 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 10 21:54:20.410411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:20.410300 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 10 21:54:20.410411 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:20.410369 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 10 21:54:21.410207 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:21.410168 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 10 21:54:21.410382 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:21.410231 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 10 21:54:22.413507 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:22.413475 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:54:22.413965 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:22.413822 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:54:22.414652 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:54:22.414632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:55:54.718236 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.718150 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd"] Jul 10 21:55:54.718693 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.718511 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:55:54.718693 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.718523 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="565e1e32-222a-493d-8a40-6a8f7a0aabaa" containerName="istio-proxy" Jul 10 21:55:54.720696 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.720677 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:54.723110 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.723056 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:55:54.729356 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.729324 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd"] Jul 10 21:55:54.816530 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.816485 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd\" (UID: \"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:54.917385 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.917342 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd\" (UID: \"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:54.918012 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:54.917985 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd\" (UID: \"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:55.032435 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:55.032344 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:55.172693 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:55.172663 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd"] Jul 10 21:55:55.175195 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:55:55.175169 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee04d77b_3667_4697_a2f4_ce8ac6a4bfff.slice/crio-75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131 WatchSource:0}: Error finding container 75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131: Status 404 returned error can't find the container with id 75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131 Jul 10 21:55:56.086322 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:56.086285 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee04d77b-3667-4697-a2f4-ce8ac6a4bfff" containerID="e91b8c9db5efa6516e0411f584d20904662aa583d8ff84a527aec9ef49d3242c" exitCode=0 Jul 10 21:55:56.086322 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:56.086319 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" event={"ID":"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff","Type":"ContainerDied","Data":"e91b8c9db5efa6516e0411f584d20904662aa583d8ff84a527aec9ef49d3242c"} Jul 10 21:55:56.086796 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:56.086369 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" event={"ID":"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff","Type":"ContainerStarted","Data":"75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131"} Jul 10 21:55:57.226960 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:57.226934 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:57.337194 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:57.337160 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca\") pod \"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff\" (UID: \"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff\") " Jul 10 21:55:57.337552 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:57.337526 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee04d77b-3667-4697-a2f4-ce8ac6a4bfff" (UID: "ee04d77b-3667-4697-a2f4-ce8ac6a4bfff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:55:57.438032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:57.437991 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee04d77b-3667-4697-a2f4-ce8ac6a4bfff-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:55:58.098371 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:58.098342 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" Jul 10 21:55:58.098566 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:58.098341 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd" event={"ID":"ee04d77b-3667-4697-a2f4-ce8ac6a4bfff","Type":"ContainerDied","Data":"75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131"} Jul 10 21:55:58.098566 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:55:58.098466 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c582bb69ffb7f22de6f19e58d201f0ebce9fcf22bd0d542836e879a3e4d131" Jul 10 21:56:02.223584 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:02.223548 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:56:02.226752 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:02.226725 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 21:56:02.230846 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:02.230820 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:56:02.233796 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:02.233773 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 21:56:20.521418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.521310 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:56:20.521901 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.521528 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" containerID="cri-o://0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96" gracePeriod=30 Jul 10 21:56:20.921638 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.921575 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp"] Jul 10 21:56:20.921969 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.921957 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee04d77b-3667-4697-a2f4-ce8ac6a4bfff" containerName="revoke-keys" Jul 10 21:56:20.922014 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.921972 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee04d77b-3667-4697-a2f4-ce8ac6a4bfff" containerName="revoke-keys" Jul 10 21:56:20.922049 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.922029 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee04d77b-3667-4697-a2f4-ce8ac6a4bfff" containerName="revoke-keys" Jul 10 21:56:20.925092 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.925047 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:20.927262 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.927233 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:56:20.932033 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:20.932002 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp"] Jul 10 21:56:21.047775 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:21.047727 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp\" (UID: \"ec0d1152-c044-46b1-bb64-a09f6c1388dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:21.149268 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:21.149223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp\" (UID: \"ec0d1152-c044-46b1-bb64-a09f6c1388dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:21.149886 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:21.149863 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp\" (UID: \"ec0d1152-c044-46b1-bb64-a09f6c1388dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:21.237401 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:21.237310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:21.383023 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:21.382991 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp"] Jul 10 21:56:21.385521 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:56:21.385489 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0d1152_c044_46b1_bb64_a09f6c1388dc.slice/crio-8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8 WatchSource:0}: Error finding container 8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8: Status 404 returned error can't find the container with id 8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8 Jul 10 21:56:22.226285 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:22.226243 2561 generic.go:356] "Generic (PLEG): container finished" podID="ec0d1152-c044-46b1-bb64-a09f6c1388dc" containerID="0df24b0a4bcf0fc37eab803336e9892ad29ef4cdcb4b41679f96ac20b77d2eac" exitCode=0 Jul 10 21:56:22.226672 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:22.226329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" event={"ID":"ec0d1152-c044-46b1-bb64-a09f6c1388dc","Type":"ContainerDied","Data":"0df24b0a4bcf0fc37eab803336e9892ad29ef4cdcb4b41679f96ac20b77d2eac"} Jul 10 21:56:22.226672 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:22.226389 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" event={"ID":"ec0d1152-c044-46b1-bb64-a09f6c1388dc","Type":"ContainerStarted","Data":"8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8"} Jul 10 21:56:22.416725 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:22.416634 2561 patch_prober.go:28] interesting pod/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 21:56:22.416725 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:22.416691 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 21:56:23.367846 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:23.367817 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:23.471338 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:23.471301 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca\") pod \"ec0d1152-c044-46b1-bb64-a09f6c1388dc\" (UID: \"ec0d1152-c044-46b1-bb64-a09f6c1388dc\") " Jul 10 21:56:23.471690 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:23.471665 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec0d1152-c044-46b1-bb64-a09f6c1388dc" (UID: "ec0d1152-c044-46b1-bb64-a09f6c1388dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:56:23.572106 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:23.571986 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0d1152-c044-46b1-bb64-a09f6c1388dc-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:24.236708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:24.236672 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" Jul 10 21:56:24.236708 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:24.236696 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp" event={"ID":"ec0d1152-c044-46b1-bb64-a09f6c1388dc","Type":"ContainerDied","Data":"8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8"} Jul 10 21:56:24.236931 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:24.236750 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7c300ef380683a5c1e280d23acb47bba4fe9a2b13b871f35887e31cc91b6e8" Jul 10 21:56:25.775454 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.775420 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:56:25.893555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893512 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893580 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893623 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893650 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893679 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmthc\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893702 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.893755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893734 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.894058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893786 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.894058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893822 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert\") pod \"03af1dba-43ef-4baf-b419-b8de686544fc\" (UID: \"03af1dba-43ef-4baf-b419-b8de686544fc\") " Jul 10 21:56:25.894058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893866 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:56:25.894058 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.893990 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:56:25.894296 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.894205 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.894296 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.894225 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.894296 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.894221 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:56:25.894421 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.894369 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:56:25.894857 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.894827 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:56:25.896208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.896177 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:56:25.896208 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.896186 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:56:25.896493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.896466 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc" (OuterVolumeSpecName: "kube-api-access-dmthc") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "kube-api-access-dmthc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:56:25.896493 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.896487 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "03af1dba-43ef-4baf-b419-b8de686544fc" (UID: "03af1dba-43ef-4baf-b419-b8de686544fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:56:25.995123 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995050 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995123 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995115 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03af1dba-43ef-4baf-b419-b8de686544fc-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995123 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995125 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995123 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995134 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03af1dba-43ef-4baf-b419-b8de686544fc-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995144 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmthc\" (UniqueName: \"kubernetes.io/projected/03af1dba-43ef-4baf-b419-b8de686544fc-kube-api-access-dmthc\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995153 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:25.995394 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:25.995163 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03af1dba-43ef-4baf-b419-b8de686544fc-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:56:26.247840 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.247805 2561 generic.go:356] "Generic (PLEG): container finished" podID="03af1dba-43ef-4baf-b419-b8de686544fc" containerID="0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96" exitCode=0 Jul 10 21:56:26.248008 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.247856 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" Jul 10 21:56:26.248008 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.247898 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerDied","Data":"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96"} Jul 10 21:56:26.248008 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.247967 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v" event={"ID":"03af1dba-43ef-4baf-b419-b8de686544fc","Type":"ContainerDied","Data":"7e0721a1543efae7ea7980dd017d504163363b43570e1bd9b418134b80a344f3"} Jul 10 21:56:26.248008 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.247996 2561 scope.go:117] "RemoveContainer" containerID="0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96" Jul 10 21:56:26.256633 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.256611 2561 scope.go:117] "RemoveContainer" containerID="43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634" Jul 10 21:56:26.265476 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.265447 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:56:26.265790 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.265774 2561 scope.go:117] "RemoveContainer" containerID="0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96" Jul 10 21:56:26.266115 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:56:26.266095 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96\": container with ID starting with 0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96 not found: ID does not exist" containerID="0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96" Jul 10 21:56:26.266181 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.266124 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96"} err="failed to get container status \"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96\": rpc error: code = NotFound desc = could not find container \"0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96\": container with ID starting with 0cfbf5eebb9a98303bd7ef6f10a1e6ae9111da4d14e0321c74f8dace47334d96 not found: ID does not exist" Jul 10 21:56:26.266181 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.266142 2561 scope.go:117] "RemoveContainer" containerID="43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634" Jul 10 21:56:26.266387 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:56:26.266370 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634\": container with ID starting with 43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634 not found: ID does not exist" containerID="43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634" Jul 10 21:56:26.266428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.266392 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634"} err="failed to get container status \"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634\": rpc error: code = NotFound desc = could not find container \"43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634\": container with ID starting with 43a726d64b5030126c365df4bd2fde0308e3863023c9fd58452305e8411ac634 not found: ID does not exist" Jul 10 21:56:26.269500 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:26.269476 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0b0348-openshift-default-54479bbd67-rws9v"] Jul 10 21:56:28.183643 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:28.183604 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" path="/var/lib/kubelet/pods/03af1dba-43ef-4baf-b419-b8de686544fc/volumes" Jul 10 21:56:47.649762 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.649724 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650258 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650280 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650299 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0d1152-c044-46b1-bb64-a09f6c1388dc" containerName="revoke-keys" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650308 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0d1152-c044-46b1-bb64-a09f6c1388dc" containerName="revoke-keys" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650320 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650329 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650440 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0d1152-c044-46b1-bb64-a09f6c1388dc" containerName="revoke-keys" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650453 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.650670 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.650464 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="03af1dba-43ef-4baf-b419-b8de686544fc" containerName="istio-proxy" Jul 10 21:56:47.653656 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.653635 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.656010 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.655988 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-205080-openshift-default-dockercfg-8mfxm\"" Jul 10 21:56:47.665645 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.665615 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:56:47.791539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791495 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791620 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791755 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791952 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791753 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791952 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c264p\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.791952 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.791813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892256 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892295 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892293 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892533 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c264p\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892533 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892533 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892518 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892580 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892613 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892648 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892683 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892679 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.892973 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.892876 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.893029 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.893014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.893428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.893403 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.895057 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.895025 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.895346 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.895324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.900276 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.900214 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.900519 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.900497 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c264p\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p\") pod \"e2e-api-a-205080-openshift-default-5c89b8b789-r58m2\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:47.964584 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:47.964541 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:48.117315 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.117274 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:56:48.118286 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:56:48.118257 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a76582_8fc5_4a2c_a8eb_14540ae3afd3.slice/crio-3f367d7c675539fa412c02cb0c07e2e29e429b692df9940660bb321f6adf4d8a WatchSource:0}: Error finding container 3f367d7c675539fa412c02cb0c07e2e29e429b692df9940660bb321f6adf4d8a: Status 404 returned error can't find the container with id 3f367d7c675539fa412c02cb0c07e2e29e429b692df9940660bb321f6adf4d8a Jul 10 21:56:48.120513 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.120472 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:56:48.120601 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.120560 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:56:48.120642 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.120603 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:56:48.357516 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.357482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerStarted","Data":"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8"} Jul 10 21:56:48.357721 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.357542 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerStarted","Data":"3f367d7c675539fa412c02cb0c07e2e29e429b692df9940660bb321f6adf4d8a"} Jul 10 21:56:48.381732 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.381661 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podStartSLOduration=1.381637584 podStartE2EDuration="1.381637584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:56:48.380005831 +0000 UTC m=+2146.730529432" watchObservedRunningTime="2026-07-10 21:56:48.381637584 +0000 UTC m=+2146.732161187" Jul 10 21:56:48.965448 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:48.965408 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:49.970348 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:49.970314 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:50.367983 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:50.367948 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:56:50.369021 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:56:50.368996 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:04.438554 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.438517 2561 generic.go:356] "Generic (PLEG): container finished" podID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerID="f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8" exitCode=0 Jul 10 21:57:04.439007 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.438587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerDied","Data":"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8"} Jul 10 21:57:04.439007 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.438998 2561 scope.go:117] "RemoveContainer" containerID="f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8" Jul 10 21:57:04.440173 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.440138 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:57:04.440243 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.440229 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:57:04.440286 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:04.440265 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:57:05.368875 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:05.368814 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:05.446509 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:05.446468 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerStarted","Data":"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880"} Jul 10 21:57:05.964937 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:05.964898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:05.976091 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:05.973028 2561 patch_prober.go:28] interesting pod/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 21:57:05.976091 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:05.973120 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 21:57:06.965979 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:06.965941 2561 patch_prober.go:28] interesting pod/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 21:57:06.966615 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:06.966578 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 21:57:07.965752 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:07.965708 2561 patch_prober.go:28] interesting pod/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 21:57:07.965984 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:07.965797 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 21:57:08.965645 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:08.965592 2561 patch_prober.go:28] interesting pod/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 21:57:08.966203 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:08.965675 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 21:57:09.969255 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:09.969223 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:09.969728 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:09.969525 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:09.970212 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:09.970197 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:12.950773 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:12.950732 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n"] Jul 10 21:57:12.954406 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:12.954385 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:12.956512 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:12.956487 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:57:12.961313 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:12.961283 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n"] Jul 10 21:57:13.038754 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.038706 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-205080-2r29n\" (UID: \"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:13.140258 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.140223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-205080-2r29n\" (UID: \"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:13.140835 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.140809 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-205080-2r29n\" (UID: \"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:13.265840 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.265750 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:13.409205 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.409175 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n"] Jul 10 21:57:13.412089 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:57:13.412028 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0909bc4a_4c5f_4ce3_a3ee_ee94a36ef79b.slice/crio-1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e WatchSource:0}: Error finding container 1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e: Status 404 returned error can't find the container with id 1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e Jul 10 21:57:13.487120 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:13.487083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" event={"ID":"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b","Type":"ContainerStarted","Data":"1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e"} Jul 10 21:57:14.494215 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:14.494103 2561 generic.go:356] "Generic (PLEG): container finished" podID="0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b" containerID="dc87772836dba0b86fe99896e50dec2cf7ee8effebc1371f1a448703c96fedc5" exitCode=0 Jul 10 21:57:14.494215 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:14.494169 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" event={"ID":"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b","Type":"ContainerDied","Data":"dc87772836dba0b86fe99896e50dec2cf7ee8effebc1371f1a448703c96fedc5"} Jul 10 21:57:15.633344 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:15.633317 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:15.762614 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:15.762509 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca\") pod \"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b\" (UID: \"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b\") " Jul 10 21:57:15.762891 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:15.762865 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b" (UID: "0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:57:15.864020 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:15.863966 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:16.506032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:16.505949 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" Jul 10 21:57:16.506032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:16.505959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-205080-2r29n" event={"ID":"0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b","Type":"ContainerDied","Data":"1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e"} Jul 10 21:57:16.506032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:16.506007 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e076bfebf3b3bb5422820b6b72cfc9246bfe8d3315233670c92f6005b46764e" Jul 10 21:57:38.704449 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:38.704409 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:57:38.704961 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:38.704689 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" containerID="cri-o://db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880" gracePeriod=30 Jul 10 21:57:39.099590 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.099495 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w"] Jul 10 21:57:39.099868 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.099855 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b" containerName="revoke-keys" Jul 10 21:57:39.099913 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.099870 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b" containerName="revoke-keys" Jul 10 21:57:39.099948 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.099924 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b" containerName="revoke-keys" Jul 10 21:57:39.103102 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.103057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:39.105291 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.105269 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 21:57:39.110792 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.110511 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w"] Jul 10 21:57:39.178333 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.178289 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85696a-gbl9w\" (UID: \"a4e1b426-6fe1-4a40-9d61-20c43d516e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:39.279769 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.279722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85696a-gbl9w\" (UID: \"a4e1b426-6fe1-4a40-9d61-20c43d516e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:39.280416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.280389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-85696a-gbl9w\" (UID: \"a4e1b426-6fe1-4a40-9d61-20c43d516e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:39.415002 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.414965 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:39.558566 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.558520 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w"] Jul 10 21:57:39.561468 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:57:39.561442 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e1b426_6fe1_4a40_9d61_20c43d516e35.slice/crio-88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7 WatchSource:0}: Error finding container 88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7: Status 404 returned error can't find the container with id 88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7 Jul 10 21:57:39.626032 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.625997 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" event={"ID":"a4e1b426-6fe1-4a40-9d61-20c43d516e35","Type":"ContainerStarted","Data":"88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7"} Jul 10 21:57:39.972171 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.972127 2561 patch_prober.go:28] interesting pod/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 21:57:39.972542 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:39.972187 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 21:57:40.632734 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:40.632696 2561 generic.go:356] "Generic (PLEG): container finished" podID="a4e1b426-6fe1-4a40-9d61-20c43d516e35" containerID="49d12c6e7528e0139c94561656e450a382956d2e35a5de2b55e0184dc96e201b" exitCode=0 Jul 10 21:57:40.632982 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:40.632786 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" event={"ID":"a4e1b426-6fe1-4a40-9d61-20c43d516e35","Type":"ContainerDied","Data":"49d12c6e7528e0139c94561656e450a382956d2e35a5de2b55e0184dc96e201b"} Jul 10 21:57:41.773891 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:41.773859 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:41.802174 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:41.802141 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca\") pod \"a4e1b426-6fe1-4a40-9d61-20c43d516e35\" (UID: \"a4e1b426-6fe1-4a40-9d61-20c43d516e35\") " Jul 10 21:57:41.802626 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:41.802595 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4e1b426-6fe1-4a40-9d61-20c43d516e35" (UID: "a4e1b426-6fe1-4a40-9d61-20c43d516e35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:57:41.903222 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:41.903124 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e1b426-6fe1-4a40-9d61-20c43d516e35-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:42.643567 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:42.643537 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" Jul 10 21:57:42.643726 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:42.643532 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-85696a-gbl9w" event={"ID":"a4e1b426-6fe1-4a40-9d61-20c43d516e35","Type":"ContainerDied","Data":"88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7"} Jul 10 21:57:42.643726 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:42.643666 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ccc9665216bd0792f31b5075d659d8c1120a9dea15b7f15c750a7fd853abe7" Jul 10 21:57:43.966881 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:43.966818 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:44.024339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024305 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024339 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024344 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024387 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024412 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024453 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c264p\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024483 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024531 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024596 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024596 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024801 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024640 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert\") pod \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\" (UID: \"27a76582-8fc5-4a2c-a8eb-14540ae3afd3\") " Jul 10 21:57:44.024801 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024674 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:57:44.024801 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024750 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:57:44.024984 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024966 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.025036 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.024990 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.025106 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.025028 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:57:44.025106 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.025029 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:57:44.025416 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.025339 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data" (OuterVolumeSpecName: "istio-data") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:57:44.026940 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.026909 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 21:57:44.027043 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.026985 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:57:44.027043 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.027005 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token" (OuterVolumeSpecName: "istio-token") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:57:44.027428 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.027408 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p" (OuterVolumeSpecName: "kube-api-access-c264p") pod "27a76582-8fc5-4a2c-a8eb-14540ae3afd3" (UID: "27a76582-8fc5-4a2c-a8eb-14540ae3afd3"). InnerVolumeSpecName "kube-api-access-c264p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:57:44.125889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125848 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.125889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125883 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c264p\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-kube-api-access-c264p\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.125889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125894 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.125889 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125903 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.126220 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125911 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.126220 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125919 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.126220 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.125926 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a76582-8fc5-4a2c-a8eb-14540ae3afd3-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 21:57:44.654854 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.654819 2561 generic.go:356] "Generic (PLEG): container finished" podID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerID="db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880" exitCode=0 Jul 10 21:57:44.655047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.654880 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" Jul 10 21:57:44.655047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.654912 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerDied","Data":"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880"} Jul 10 21:57:44.655047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.654968 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2" event={"ID":"27a76582-8fc5-4a2c-a8eb-14540ae3afd3","Type":"ContainerDied","Data":"3f367d7c675539fa412c02cb0c07e2e29e429b692df9940660bb321f6adf4d8a"} Jul 10 21:57:44.655047 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.654991 2561 scope.go:117] "RemoveContainer" containerID="db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880" Jul 10 21:57:44.664446 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.664425 2561 scope.go:117] "RemoveContainer" containerID="f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8" Jul 10 21:57:44.673006 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.672976 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:57:44.673797 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.673773 2561 scope.go:117] "RemoveContainer" containerID="db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880" Jul 10 21:57:44.674142 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:57:44.674119 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880\": container with ID starting with db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880 not found: ID does not exist" containerID="db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880" Jul 10 21:57:44.674221 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.674154 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880"} err="failed to get container status \"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880\": rpc error: code = NotFound desc = could not find container \"db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880\": container with ID starting with db4a5d53a99ef3e44987d8b07dd1fe0b7371316bae6cf0fc742524f9dfc1e880 not found: ID does not exist" Jul 10 21:57:44.674221 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.674174 2561 scope.go:117] "RemoveContainer" containerID="f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8" Jul 10 21:57:44.674525 ip-10-0-130-213 kubenswrapper[2561]: E0710 21:57:44.674498 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8\": container with ID starting with f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8 not found: ID does not exist" containerID="f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8" Jul 10 21:57:44.674685 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.674534 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8"} err="failed to get container status \"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8\": rpc error: code = NotFound desc = could not find container \"f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8\": container with ID starting with f562992bec4f171ee28f153d7671823484d065751bd4c1fd3b2f7a0065c516e8 not found: ID does not exist" Jul 10 21:57:44.677389 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:44.677366 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-205080-openshift-default-5c89b8b789-r58m2"] Jul 10 21:57:46.184364 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:57:46.184329 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" path="/var/lib/kubelet/pods/27a76582-8fc5-4a2c-a8eb-14540ae3afd3/volumes" Jul 10 21:58:05.744138 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744099 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744435 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744446 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744457 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744463 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744474 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e1b426-6fe1-4a40-9d61-20c43d516e35" containerName="revoke-keys" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744479 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e1b426-6fe1-4a40-9d61-20c43d516e35" containerName="revoke-keys" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744552 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e1b426-6fe1-4a40-9d61-20c43d516e35" containerName="revoke-keys" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744563 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.744659 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.744570 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a76582-8fc5-4a2c-a8eb-14540ae3afd3" containerName="istio-proxy" Jul 10 21:58:05.747860 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.747834 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.750051 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.750026 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-22d37c-openshift-default-dockercfg-tvzsh\"" Jul 10 21:58:05.759418 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.759391 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 21:58:05.822274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822274 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822275 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822382 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822488 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7tm\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822618 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822618 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822541 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.822618 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.822559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924082 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924124 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924172 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924194 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924224 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924248 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7tm\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924302 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924612 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924612 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924536 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924612 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924612 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.924938 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924911 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.925012 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.924945 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.925301 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.925280 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.926828 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.926807 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.927119 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.927097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.932468 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.932440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:05.932555 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:05.932527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7tm\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm\") pod \"e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:06.059981 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.059882 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:06.207614 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.207587 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 21:58:06.209792 ip-10-0-130-213 kubenswrapper[2561]: W0710 21:58:06.209760 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d1cc1c_c24c_4efe_9b5e_68cc9d8c964e.slice/crio-030cac5576451c2dc6b3228b14bcb7835dcc8ce925d5e50740edb8a33049535e WatchSource:0}: Error finding container 030cac5576451c2dc6b3228b14bcb7835dcc8ce925d5e50740edb8a33049535e: Status 404 returned error can't find the container with id 030cac5576451c2dc6b3228b14bcb7835dcc8ce925d5e50740edb8a33049535e Jul 10 21:58:06.212456 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.212418 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:06.212539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.212505 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:06.212539 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.212533 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:06.770979 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.770938 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerStarted","Data":"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11"} Jul 10 21:58:06.771383 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.771003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerStarted","Data":"030cac5576451c2dc6b3228b14bcb7835dcc8ce925d5e50740edb8a33049535e"} Jul 10 21:58:06.792357 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:06.792300 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podStartSLOduration=1.792282621 podStartE2EDuration="1.792282621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:58:06.791022213 +0000 UTC m=+2225.141545794" watchObservedRunningTime="2026-07-10 21:58:06.792282621 +0000 UTC m=+2225.142806268" Jul 10 21:58:07.060815 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:07.060719 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:07.065606 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:07.065576 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:07.776046 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:07.776014 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:07.777128 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:07.777109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:22.778103 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.777969 2561 patch_prober.go:28] interesting pod/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 10 21:58:22.778508 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.778098 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 10 21:58:22.855133 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.855093 2561 generic.go:356] "Generic (PLEG): container finished" podID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerID="6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11" exitCode=0 Jul 10 21:58:22.855305 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.855170 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerDied","Data":"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11"} Jul 10 21:58:22.855652 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.855638 2561 scope.go:117] "RemoveContainer" containerID="6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11" Jul 10 21:58:22.856676 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.856647 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:22.856783 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.856717 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:22.856783 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:22.856749 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 21:58:23.864894 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:23.864855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerStarted","Data":"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0"} Jul 10 21:58:24.060644 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:24.060596 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:24.060861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:24.060750 2561 patch_prober.go:28] interesting pod/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 10 21:58:24.060861 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:24.060813 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 10 21:58:25.061414 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:25.061365 2561 patch_prober.go:28] interesting pod/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 10 21:58:25.061920 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:25.061440 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 10 21:58:26.061165 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:26.061117 2561 patch_prober.go:28] interesting pod/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 10 21:58:26.061392 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:26.061190 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 10 21:58:27.061182 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:27.061140 2561 patch_prober.go:28] interesting pod/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 10 21:58:27.061688 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:27.061230 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 10 21:58:28.065102 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:28.065055 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:28.065578 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:28.065557 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 21:58:28.066293 ip-10-0-130-213 kubenswrapper[2561]: I0710 21:58:28.066274 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 22:00:00.141602 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.141518 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp"] Jul 10 22:00:00.145308 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.145283 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752"] Jul 10 22:00:00.145473 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.145412 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" Jul 10 22:00:00.147611 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.147588 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 22:00:00.148476 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.148457 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch"] Jul 10 22:00:00.148603 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.148582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" Jul 10 22:00:00.151650 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.151627 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" Jul 10 22:00:00.154276 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.154253 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp"] Jul 10 22:00:00.158293 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.158260 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" Jul 10 22:00:00.160612 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.160587 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752"] Jul 10 22:00:00.169054 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.169029 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" Jul 10 22:00:00.175595 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.175559 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch"] Jul 10 22:00:00.205421 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.205390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" Jul 10 22:00:00.345547 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.345508 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp"] Jul 10 22:00:00.347792 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:00:00.347759 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1730ab_8d6e_44fb_8539_28204dedd8bd.slice/crio-9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2 WatchSource:0}: Error finding container 9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2: Status 404 returned error can't find the container with id 9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2 Jul 10 22:00:00.366030 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.365906 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" event={"ID":"2a1730ab-8d6e-44fb-8539-28204dedd8bd","Type":"ContainerStarted","Data":"9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2"} Jul 10 22:00:00.366185 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.366108 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752"] Jul 10 22:00:00.369483 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:00:00.369452 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808188dd_5934_4314_99d3_41d3a338ff02.slice/crio-41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284 WatchSource:0}: Error finding container 41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284: Status 404 returned error can't find the container with id 41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284 Jul 10 22:00:00.398802 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:00.398760 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch"] Jul 10 22:00:00.401604 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:00:00.401574 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922c8680_030b_42fd_bcda_70cb5e98932e.slice/crio-4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a WatchSource:0}: Error finding container 4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a: Status 404 returned error can't find the container with id 4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a Jul 10 22:00:01.373488 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.373451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" event={"ID":"922c8680-030b-42fd-bcda-70cb5e98932e","Type":"ContainerStarted","Data":"03b32623c81c9a2ddf0d4bedbad3b0e3142792dcd2a795c236627a7ca43a2e6e"} Jul 10 22:00:01.373934 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.373516 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" event={"ID":"922c8680-030b-42fd-bcda-70cb5e98932e","Type":"ContainerStarted","Data":"4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a"} Jul 10 22:00:01.374902 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.374876 2561 generic.go:356] "Generic (PLEG): container finished" podID="2a1730ab-8d6e-44fb-8539-28204dedd8bd" containerID="e46c8dc239be3fd08faea5de3cf5cc089e4c45cf18954234cd0110ebaca69ba8" exitCode=0 Jul 10 22:00:01.375025 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.374940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" event={"ID":"2a1730ab-8d6e-44fb-8539-28204dedd8bd","Type":"ContainerDied","Data":"e46c8dc239be3fd08faea5de3cf5cc089e4c45cf18954234cd0110ebaca69ba8"} Jul 10 22:00:01.376366 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.376342 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" event={"ID":"808188dd-5934-4314-99d3-41d3a338ff02","Type":"ContainerStarted","Data":"60aa4017aa3db2ff734a14b1fbdc9c9d83b563c5ab3ed58792c9fc34185050b2"} Jul 10 22:00:01.376465 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.376386 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" event={"ID":"808188dd-5934-4314-99d3-41d3a338ff02","Type":"ContainerStarted","Data":"41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284"} Jul 10 22:00:01.389846 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.389793 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" podStartSLOduration=1.3897774520000001 podStartE2EDuration="1.389777452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:00:01.388452579 +0000 UTC m=+2339.738976156" watchObservedRunningTime="2026-07-10 22:00:01.389777452 +0000 UTC m=+2339.740301007" Jul 10 22:00:01.419129 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:01.419028 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" podStartSLOduration=1.4190104510000001 podStartE2EDuration="1.419010451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:00:01.417226483 +0000 UTC m=+2339.767987903" watchObservedRunningTime="2026-07-10 22:00:01.419010451 +0000 UTC m=+2339.769534029" Jul 10 22:00:02.384597 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:02.384559 2561 generic.go:356] "Generic (PLEG): container finished" podID="922c8680-030b-42fd-bcda-70cb5e98932e" containerID="03b32623c81c9a2ddf0d4bedbad3b0e3142792dcd2a795c236627a7ca43a2e6e" exitCode=0 Jul 10 22:00:02.385027 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:02.384645 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" event={"ID":"922c8680-030b-42fd-bcda-70cb5e98932e","Type":"ContainerDied","Data":"03b32623c81c9a2ddf0d4bedbad3b0e3142792dcd2a795c236627a7ca43a2e6e"} Jul 10 22:00:02.386204 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:02.386177 2561 generic.go:356] "Generic (PLEG): container finished" podID="808188dd-5934-4314-99d3-41d3a338ff02" containerID="60aa4017aa3db2ff734a14b1fbdc9c9d83b563c5ab3ed58792c9fc34185050b2" exitCode=0 Jul 10 22:00:02.386307 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:02.386251 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" event={"ID":"808188dd-5934-4314-99d3-41d3a338ff02","Type":"ContainerDied","Data":"60aa4017aa3db2ff734a14b1fbdc9c9d83b563c5ab3ed58792c9fc34185050b2"} Jul 10 22:00:02.529825 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:02.529796 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" Jul 10 22:00:03.393139 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:03.393097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" event={"ID":"2a1730ab-8d6e-44fb-8539-28204dedd8bd","Type":"ContainerDied","Data":"9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2"} Jul 10 22:00:03.393139 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:03.393130 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728680-hrtxp" Jul 10 22:00:03.393614 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:03.393157 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9585611528c45a84314ff777754b30a13e1659e26b9f1fa1cf72341cf608dab2" Jul 10 22:00:03.535110 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:03.535077 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" Jul 10 22:00:03.561922 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:03.561890 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" Jul 10 22:00:04.400006 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.399977 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" Jul 10 22:00:04.400500 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.400021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752" event={"ID":"808188dd-5934-4314-99d3-41d3a338ff02","Type":"ContainerDied","Data":"41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284"} Jul 10 22:00:04.400500 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.400100 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a6b3a2d2adf6326e61b424d2ffe91a21802dc1e371565027cbce8b97612284" Jul 10 22:00:04.401506 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.401483 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" event={"ID":"922c8680-030b-42fd-bcda-70cb5e98932e","Type":"ContainerDied","Data":"4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a"} Jul 10 22:00:04.401618 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.401517 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch" Jul 10 22:00:04.401618 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:00:04.401524 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df1b22024763eb4342b46c756e8c58df4a6e6218c160115d03ff420ac6a958a" Jul 10 22:01:02.253663 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:01:02.253629 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 22:01:02.257667 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:01:02.257641 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 22:01:02.261143 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:01:02.261120 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 22:01:02.264930 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:01:02.264911 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 22:04:32.689660 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689579 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb"] Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689909 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1730ab-8d6e-44fb-8539-28204dedd8bd" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689922 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1730ab-8d6e-44fb-8539-28204dedd8bd" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689937 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922c8680-030b-42fd-bcda-70cb5e98932e" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689943 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="922c8680-030b-42fd-bcda-70cb5e98932e" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689955 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808188dd-5934-4314-99d3-41d3a338ff02" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.689961 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="808188dd-5934-4314-99d3-41d3a338ff02" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.690011 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="922c8680-030b-42fd-bcda-70cb5e98932e" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.690018 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="808188dd-5934-4314-99d3-41d3a338ff02" containerName="cleanup" Jul 10 22:04:32.690104 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.690023 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1730ab-8d6e-44fb-8539-28204dedd8bd" containerName="cleanup" Jul 10 22:04:32.692895 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.692877 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:32.695280 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.695262 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 22:04:32.700026 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.700003 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb"] Jul 10 22:04:32.804505 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.804470 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb\" (UID: \"87182d6a-e7e7-444a-8a83-981920f68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:32.905709 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.905667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb\" (UID: \"87182d6a-e7e7-444a-8a83-981920f68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:32.906343 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:32.906322 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb\" (UID: \"87182d6a-e7e7-444a-8a83-981920f68fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:33.005022 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:33.004935 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:33.148441 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:33.148404 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb"] Jul 10 22:04:33.150225 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:04:33.150119 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87182d6a_e7e7_444a_8a83_981920f68fdb.slice/crio-cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5 WatchSource:0}: Error finding container cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5: Status 404 returned error can't find the container with id cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5 Jul 10 22:04:33.805025 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:33.804932 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" event={"ID":"87182d6a-e7e7-444a-8a83-981920f68fdb","Type":"ContainerStarted","Data":"db366768c04cf444852206c9e495f16e7c43c96040b8cfdae2ee6e10018b72bc"} Jul 10 22:04:33.805025 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:33.804990 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" event={"ID":"87182d6a-e7e7-444a-8a83-981920f68fdb","Type":"ContainerStarted","Data":"cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5"} Jul 10 22:04:33.823117 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:33.823046 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" podStartSLOduration=1.823028915 podStartE2EDuration="1.823028915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:04:33.8203322 +0000 UTC m=+2612.170855810" watchObservedRunningTime="2026-07-10 22:04:33.823028915 +0000 UTC m=+2612.173552511" Jul 10 22:04:34.812169 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:34.812137 2561 generic.go:356] "Generic (PLEG): container finished" podID="87182d6a-e7e7-444a-8a83-981920f68fdb" containerID="db366768c04cf444852206c9e495f16e7c43c96040b8cfdae2ee6e10018b72bc" exitCode=0 Jul 10 22:04:34.812569 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:34.812187 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" event={"ID":"87182d6a-e7e7-444a-8a83-981920f68fdb","Type":"ContainerDied","Data":"db366768c04cf444852206c9e495f16e7c43c96040b8cfdae2ee6e10018b72bc"} Jul 10 22:04:35.953974 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:35.953945 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:36.032351 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.032307 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca\") pod \"87182d6a-e7e7-444a-8a83-981920f68fdb\" (UID: \"87182d6a-e7e7-444a-8a83-981920f68fdb\") " Jul 10 22:04:36.032681 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.032657 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87182d6a-e7e7-444a-8a83-981920f68fdb" (UID: "87182d6a-e7e7-444a-8a83-981920f68fdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:04:36.133097 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.133032 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87182d6a-e7e7-444a-8a83-981920f68fdb-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:04:36.827474 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.827430 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" event={"ID":"87182d6a-e7e7-444a-8a83-981920f68fdb","Type":"ContainerDied","Data":"cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5"} Jul 10 22:04:36.827474 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.827479 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab26fcd5393c606f805e1eefa48d151b6e598f1fd7617b6381d701eef25d6d5" Jul 10 22:04:36.827699 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:36.827445 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb" Jul 10 22:04:43.152261 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:43.152199 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752"] Jul 10 22:04:43.154314 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:43.154287 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-22d37c-29728680-cr752"] Jul 10 22:04:44.184116 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:44.184082 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808188dd-5934-4314-99d3-41d3a338ff02" path="/var/lib/kubelet/pods/808188dd-5934-4314-99d3-41d3a338ff02/volumes" Jul 10 22:04:58.475439 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.475404 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 22:04:58.475952 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.475629 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" containerID="cri-o://cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0" gracePeriod=30 Jul 10 22:04:58.862112 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.862071 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2"] Jul 10 22:04:58.862449 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.862436 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87182d6a-e7e7-444a-8a83-981920f68fdb" containerName="revoke-keys" Jul 10 22:04:58.862493 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.862452 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87182d6a-e7e7-444a-8a83-981920f68fdb" containerName="revoke-keys" Jul 10 22:04:58.862538 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.862527 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87182d6a-e7e7-444a-8a83-981920f68fdb" containerName="revoke-keys" Jul 10 22:04:58.866037 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.866018 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:04:58.868836 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.868815 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6nw2b\"" Jul 10 22:04:58.875750 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:58.875722 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2"] Jul 10 22:04:59.036850 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.036804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2\" (UID: \"70c943a2-5346-499a-adc0-8f18c4d0a034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:04:59.137472 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.137383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2\" (UID: \"70c943a2-5346-499a-adc0-8f18c4d0a034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:04:59.137982 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.137963 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2\" (UID: \"70c943a2-5346-499a-adc0-8f18c4d0a034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:04:59.177362 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.177314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:04:59.317603 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.317556 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2"] Jul 10 22:04:59.319891 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:04:59.319861 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c943a2_5346_499a_adc0_8f18c4d0a034.slice/crio-0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267 WatchSource:0}: Error finding container 0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267: Status 404 returned error can't find the container with id 0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267 Jul 10 22:04:59.956455 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.956412 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" event={"ID":"70c943a2-5346-499a-adc0-8f18c4d0a034","Type":"ContainerStarted","Data":"14df4f54f9fb7d1c056f703e80cb797b6091b9899abf11545a01659d466bc992"} Jul 10 22:04:59.956847 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.956504 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" event={"ID":"70c943a2-5346-499a-adc0-8f18c4d0a034","Type":"ContainerStarted","Data":"0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267"} Jul 10 22:04:59.976523 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:04:59.976470 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" podStartSLOduration=1.976451524 podStartE2EDuration="1.976451524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:04:59.973776417 +0000 UTC m=+2638.324300019" watchObservedRunningTime="2026-07-10 22:04:59.976451524 +0000 UTC m=+2638.326975143" Jul 10 22:05:00.963310 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:00.963277 2561 generic.go:356] "Generic (PLEG): container finished" podID="70c943a2-5346-499a-adc0-8f18c4d0a034" containerID="14df4f54f9fb7d1c056f703e80cb797b6091b9899abf11545a01659d466bc992" exitCode=0 Jul 10 22:05:00.963698 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:00.963343 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" event={"ID":"70c943a2-5346-499a-adc0-8f18c4d0a034","Type":"ContainerDied","Data":"14df4f54f9fb7d1c056f703e80cb797b6091b9899abf11545a01659d466bc992"} Jul 10 22:05:02.101994 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.101969 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:05:02.265433 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.265331 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca\") pod \"70c943a2-5346-499a-adc0-8f18c4d0a034\" (UID: \"70c943a2-5346-499a-adc0-8f18c4d0a034\") " Jul 10 22:05:02.265734 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.265705 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70c943a2-5346-499a-adc0-8f18c4d0a034" (UID: "70c943a2-5346-499a-adc0-8f18c4d0a034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:05:02.366552 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.366510 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70c943a2-5346-499a-adc0-8f18c4d0a034-maas-api-service-ca\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:02.975934 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.975907 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" Jul 10 22:05:02.976145 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.975906 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2" event={"ID":"70c943a2-5346-499a-adc0-8f18c4d0a034","Type":"ContainerDied","Data":"0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267"} Jul 10 22:05:02.976145 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:02.976038 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b07b62ce0ab1b5cd3b318551192b46f470e6abb17f65bb0382dab973132a267" Jul 10 22:05:03.727278 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.727253 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 22:05:03.781987 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.781882 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.781987 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.781931 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.781987 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.781954 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.781987 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.781979 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl7tm\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.782338 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.782006 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.782338 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.782026 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.782338 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.782268 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:05:03.782452 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.782356 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:05:03.782529 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.782498 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:05:03.784440 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.784409 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm" (OuterVolumeSpecName: "kube-api-access-gl7tm") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "kube-api-access-gl7tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:05:03.784543 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.784445 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 22:05:03.784543 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.784471 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:05:03.883149 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883055 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.883356 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883186 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.883356 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883220 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token\") pod \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\" (UID: \"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e\") " Jul 10 22:05:03.883507 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883474 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:05:03.883577 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883491 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-envoy\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883577 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883545 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-certs\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883577 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883571 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-workload-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883694 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883586 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-podinfo\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883694 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883602 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl7tm\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-kube-api-access-gl7tm\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883694 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883618 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istiod-ca-cert\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.883965 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.883941 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data" (OuterVolumeSpecName: "istio-data") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:05:03.885445 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.885425 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token" (OuterVolumeSpecName: "istio-token") pod "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" (UID: "78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:05:03.982793 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.982757 2561 generic.go:356] "Generic (PLEG): container finished" podID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerID="cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0" exitCode=0 Jul 10 22:05:03.982984 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.982813 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerDied","Data":"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0"} Jul 10 22:05:03.982984 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.982824 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" Jul 10 22:05:03.982984 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.982878 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt" event={"ID":"78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e","Type":"ContainerDied","Data":"030cac5576451c2dc6b3228b14bcb7835dcc8ce925d5e50740edb8a33049535e"} Jul 10 22:05:03.982984 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.982901 2561 scope.go:117] "RemoveContainer" containerID="cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0" Jul 10 22:05:03.984054 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.984035 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-credential-socket\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.984214 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.984082 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-token\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.984214 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.984092 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e-istio-data\") on node \"ip-10-0-130-213.ec2.internal\" DevicePath \"\"" Jul 10 22:05:03.992262 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:03.992246 2561 scope.go:117] "RemoveContainer" containerID="6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11" Jul 10 22:05:04.000724 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.000708 2561 scope.go:117] "RemoveContainer" containerID="cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0" Jul 10 22:05:04.000981 ip-10-0-130-213 kubenswrapper[2561]: E0710 22:05:04.000959 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0\": container with ID starting with cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0 not found: ID does not exist" containerID="cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0" Jul 10 22:05:04.001030 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.000990 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0"} err="failed to get container status \"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0\": rpc error: code = NotFound desc = could not find container \"cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0\": container with ID starting with cdf06bd40adec334e17a991edd6a9db3b3669d3b3a2f82283dd7446a4bde5fa0 not found: ID does not exist" Jul 10 22:05:04.001030 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.001013 2561 scope.go:117] "RemoveContainer" containerID="6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11" Jul 10 22:05:04.001267 ip-10-0-130-213 kubenswrapper[2561]: E0710 22:05:04.001249 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11\": container with ID starting with 6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11 not found: ID does not exist" containerID="6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11" Jul 10 22:05:04.001313 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.001273 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11"} err="failed to get container status \"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11\": rpc error: code = NotFound desc = could not find container \"6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11\": container with ID starting with 6107c983bcc24b4bbb38e5693ddabe2422119e0cb18b94396f921f97a906ab11 not found: ID does not exist" Jul 10 22:05:04.005945 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.005925 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 22:05:04.012100 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.012048 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-22d37c-openshift-default-69684f4f8c-4g2dt"] Jul 10 22:05:04.183909 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:04.183879 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" path="/var/lib/kubelet/pods/78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e/volumes" Jul 10 22:05:09.343205 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:09.343164 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch"] Jul 10 22:05:09.350820 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:09.350787 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f937a7-29728680-wxzch"] Jul 10 22:05:10.184281 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:10.184248 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922c8680-030b-42fd-bcda-70cb5e98932e" path="/var/lib/kubelet/pods/922c8680-030b-42fd-bcda-70cb5e98932e/volumes" Jul 10 22:05:31.033009 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:31.032977 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d8977cb58-5l9vp_1ca37d00-d45d-4fb3-9d26-5630b7d6aece/copy-manifests/0.log" Jul 10 22:05:31.039952 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:31.039925 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d8977cb58-5l9vp_1ca37d00-d45d-4fb3-9d26-5630b7d6aece/manager/0.log" Jul 10 22:05:31.629126 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:31.629096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728665-crkrl_1223dc10-01df-402d-88fe-a09cb530b790/cleanup/0.log" Jul 10 22:05:31.743451 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:31.743424 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728680-hrtxp_2a1730ab-8d6e-44fb-8539-28204dedd8bd/cleanup/0.log" Jul 10 22:05:31.964702 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:31.964622 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-e5f00709-9qxbj_abde8d2b-ca6b-42f8-8767-c808e7564c69/revoke-keys/0.log" Jul 10 22:05:32.074303 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.074278 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-205080-2r29n_0909bc4a-4c5f-4ce3-a3ee-ee94a36ef79b/revoke-keys/0.log" Jul 10 22:05:32.186200 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.186167 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-85696a-gbl9w_a4e1b426-6fe1-4a40-9d61-20c43d516e35/revoke-keys/0.log" Jul 10 22:05:32.291696 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.291612 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-2fe53472-w92pz_d7958275-9f52-45a3-8a9b-8674e430aef6/revoke-keys/0.log" Jul 10 22:05:32.401004 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.400977 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-0b0348-fnrrd_ee04d77b-3667-4697-a2f4-ce8ac6a4bfff/revoke-keys/0.log" Jul 10 22:05:32.513663 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.513634 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-7d633a-m7lsp_ec0d1152-c044-46b1-bb64-a09f6c1388dc/revoke-keys/0.log" Jul 10 22:05:32.624053 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.624026 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-acd5952a-2rpzn_d7567810-9346-4f1c-8a39-b639e95ecb2a/revoke-keys/0.log" Jul 10 22:05:32.736867 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.736836 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-fa2f395f-q8h8z_a0b1105d-1c52-40e8-a636-c35bc2b9faef/revoke-keys/0.log" Jul 10 22:05:32.848262 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.848233 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-22d37c-tlmsb_87182d6a-e7e7-444a-8a83-981920f68fdb/revoke-keys/0.log" Jul 10 22:05:32.958935 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:32.958853 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-f937a7-5zwq2_70c943a2-5346-499a-adc0-8f18c4d0a034/revoke-keys/0.log" Jul 10 22:05:33.071703 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:33.071668 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fz8hk_3cba446f-4f19-458c-8fc8-66bbd1306c71/postgres/0.log" Jul 10 22:05:35.585337 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:35.585303 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vt9b7_8df4b9d9-8f7f-4211-b328-e61375820b42/istio-proxy/0.log" Jul 10 22:05:35.708511 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:35.708479 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7mfz7_d6b9fa2c-57db-4859-bd54-72efa3c9a00d/discovery/0.log" Jul 10 22:05:36.409024 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:36.408992 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f44597c6-hzxv6_32d29f1c-89cd-42f6-b9b6-e08b28e47266/router/0.log" Jul 10 22:05:36.884480 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:36.884449 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-r5gpx_88bb4bae-7ce5-450b-aa5b-22319d499f83/main/0.log" Jul 10 22:05:37.120765 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:37.120735 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbckpw6_bc7f4542-28f7-4ae6-88ca-e58132a0f015/main/0.log" Jul 10 22:05:45.426971 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:45.426934 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-989qr_b64784df-28d5-4bb5-b363-9169dd0308af/global-pull-secret-syncer/0.log" Jul 10 22:05:45.618908 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:45.618874 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rcbgx_3cb78a23-940e-4a6c-83bf-c0dba9254184/konnectivity-agent/0.log" Jul 10 22:05:45.640653 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:45.640627 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-213.ec2.internal_c8ec75a18410aaf8a2d5adb81c2eec0a/haproxy/0.log" Jul 10 22:05:53.478100 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:53.478050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-s7sww_4fb4b267-19c8-4f61-b649-35dfff7bec04/cluster-monitoring-operator/0.log" Jul 10 22:05:53.812454 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:53.812364 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9hh7_b7caf5ef-960a-41c9-9b50-872641002718/node-exporter/0.log" Jul 10 22:05:53.840145 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:53.840111 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9hh7_b7caf5ef-960a-41c9-9b50-872641002718/kube-rbac-proxy/0.log" Jul 10 22:05:53.869656 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:53.869626 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9hh7_b7caf5ef-960a-41c9-9b50-872641002718/init-textfile/0.log" Jul 10 22:05:55.601884 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.601844 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n"] Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602204 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602219 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602230 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602236 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602243 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c943a2-5346-499a-adc0-8f18c4d0a034" containerName="revoke-keys" Jul 10 22:05:55.602269 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602249 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c943a2-5346-499a-adc0-8f18c4d0a034" containerName="revoke-keys" Jul 10 22:05:55.602480 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602319 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c943a2-5346-499a-adc0-8f18c4d0a034" containerName="revoke-keys" Jul 10 22:05:55.602480 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602326 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.602480 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.602335 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d1cc1c-c24c-4efe-9b5e-68cc9d8c964e" containerName="istio-proxy" Jul 10 22:05:55.605504 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.605485 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.608017 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.607997 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jk69p\"/\"default-dockercfg-gpq7z\"" Jul 10 22:05:55.608177 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.607999 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jk69p\"/\"openshift-service-ca.crt\"" Jul 10 22:05:55.608706 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.608693 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jk69p\"/\"kube-root-ca.crt\"" Jul 10 22:05:55.617622 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.617594 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n"] Jul 10 22:05:55.632618 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.632589 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-lib-modules\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.632814 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.632633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2n2\" (UniqueName: \"kubernetes.io/projected/b9b9aabf-659e-482f-b633-33e2885ce794-kube-api-access-sl2n2\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.632814 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.632661 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-sys\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.632814 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.632676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-podres\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.632814 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.632804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-proc\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734086 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2n2\" (UniqueName: \"kubernetes.io/projected/b9b9aabf-659e-482f-b633-33e2885ce794-kube-api-access-sl2n2\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734255 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-sys\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734255 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734136 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-podres\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734255 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734229 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-proc\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734255 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734239 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-sys\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734402 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-lib-modules\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734402 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734316 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-proc\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734402 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734334 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-podres\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.734402 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.734342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9b9aabf-659e-482f-b633-33e2885ce794-lib-modules\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.743079 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.743037 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2n2\" (UniqueName: \"kubernetes.io/projected/b9b9aabf-659e-482f-b633-33e2885ce794-kube-api-access-sl2n2\") pod \"perf-node-gather-daemonset-2r99n\" (UID: \"b9b9aabf-659e-482f-b633-33e2885ce794\") " pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:55.771717 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.771683 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-m5bj4_ac6d2700-7161-4eea-a941-329fa08026ac/networking-console-plugin/0.log" Jul 10 22:05:55.915078 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:55.915029 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:56.051702 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.051664 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n"] Jul 10 22:05:56.053551 ip-10-0-130-213 kubenswrapper[2561]: W0710 22:05:56.053518 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb9b9aabf_659e_482f_b633_33e2885ce794.slice/crio-be6d2494cd118354d595649089b5ceae6a8c3c4cae3525666b5e435d29ce263b WatchSource:0}: Error finding container be6d2494cd118354d595649089b5ceae6a8c3c4cae3525666b5e435d29ce263b: Status 404 returned error can't find the container with id be6d2494cd118354d595649089b5ceae6a8c3c4cae3525666b5e435d29ce263b Jul 10 22:05:56.251164 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.251120 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" event={"ID":"b9b9aabf-659e-482f-b633-33e2885ce794","Type":"ContainerStarted","Data":"52dc0dfc1061993f56ad350ba91e3c58d1499022105cdbbbc6fe67d9390027fd"} Jul 10 22:05:56.251348 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.251181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" event={"ID":"b9b9aabf-659e-482f-b633-33e2885ce794","Type":"ContainerStarted","Data":"be6d2494cd118354d595649089b5ceae6a8c3c4cae3525666b5e435d29ce263b"} Jul 10 22:05:56.251348 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.251261 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:05:56.269806 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.269748 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" podStartSLOduration=1.269731693 podStartE2EDuration="1.269731693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:05:56.267791832 +0000 UTC m=+2694.618315410" watchObservedRunningTime="2026-07-10 22:05:56.269731693 +0000 UTC m=+2694.620255272" Jul 10 22:05:56.374729 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.374669 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 22:05:56.382935 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:56.382901 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/2.log" Jul 10 22:05:57.595510 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:57.595481 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-hj44q_92294bbd-9727-4430-9c3b-875d7c67e510/volume-data-source-validator/0.log" Jul 10 22:05:58.836279 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:58.836252 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x8k5_acb3f997-86fd-499f-9b1b-f9b657fbfdac/dns/0.log" Jul 10 22:05:58.860458 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:58.860423 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x8k5_acb3f997-86fd-499f-9b1b-f9b657fbfdac/kube-rbac-proxy/0.log" Jul 10 22:05:58.939129 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:58.939101 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-649j8_fc8123ab-1603-45d5-aeef-f5ccf4f8c2bc/dns-node-resolver/0.log" Jul 10 22:05:59.716889 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:05:59.716857 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cq2g5_3229741a-df29-40c4-9abe-18205becc9d0/node-ca/0.log" Jul 10 22:06:00.738621 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:00.738592 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vt9b7_8df4b9d9-8f7f-4211-b328-e61375820b42/istio-proxy/0.log" Jul 10 22:06:00.944958 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:00.944926 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7mfz7_d6b9fa2c-57db-4859-bd54-72efa3c9a00d/discovery/0.log" Jul 10 22:06:01.201856 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:01.201816 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f44597c6-hzxv6_32d29f1c-89cd-42f6-b9b6-e08b28e47266/router/0.log" Jul 10 22:06:01.965209 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:01.965164 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5t7db_a256bf08-cc43-4cae-ade0-910b4772530d/serve-healthcheck-canary/0.log" Jul 10 22:06:02.266782 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.266694 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jk69p/perf-node-gather-daemonset-2r99n" Jul 10 22:06:02.289141 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.289117 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 22:06:02.292801 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.292772 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-d642m_f47a0d11-6bb6-4368-812b-c41aa6b2ccbb/console-operator/1.log" Jul 10 22:06:02.296310 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.296283 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 22:06:02.300077 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.300039 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t9277_6087ea90-6a0f-4b14-b524-5808dce10ed5/ovn-acl-logging/0.log" Jul 10 22:06:02.427027 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.426998 2561 scope.go:117] "RemoveContainer" containerID="03b32623c81c9a2ddf0d4bedbad3b0e3142792dcd2a795c236627a7ca43a2e6e" Jul 10 22:06:02.435329 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.435303 2561 scope.go:117] "RemoveContainer" containerID="60aa4017aa3db2ff734a14b1fbdc9c9d83b563c5ab3ed58792c9fc34185050b2" Jul 10 22:06:02.546834 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.546738 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qqx7l_de2aecbb-9a62-48c4-887b-92b03ac8392b/insights-operator/0.log" Jul 10 22:06:02.550362 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.550337 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qqx7l_de2aecbb-9a62-48c4-887b-92b03ac8392b/insights-operator/1.log" Jul 10 22:06:02.758552 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.758524 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnhh5_b1ca98a1-b383-43b2-b060-b33658e0b5aa/kube-rbac-proxy/0.log" Jul 10 22:06:02.782680 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.782649 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnhh5_b1ca98a1-b383-43b2-b060-b33658e0b5aa/exporter/0.log" Jul 10 22:06:02.806154 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:02.806046 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnhh5_b1ca98a1-b383-43b2-b060-b33658e0b5aa/extractor/0.log" Jul 10 22:06:05.703844 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:05.703811 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d8977cb58-5l9vp_1ca37d00-d45d-4fb3-9d26-5630b7d6aece/manager/0.log" Jul 10 22:06:05.728879 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:05.728851 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d8977cb58-5l9vp_1ca37d00-d45d-4fb3-9d26-5630b7d6aece/copy-manifests/0.log" Jul 10 22:06:05.748679 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:05.748648 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vt9b7_8df4b9d9-8f7f-4211-b328-e61375820b42/istio-proxy/0.log" Jul 10 22:06:05.955965 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:05.955863 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7mfz7_d6b9fa2c-57db-4859-bd54-72efa3c9a00d/discovery/0.log" Jul 10 22:06:06.155939 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:06.155907 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f44597c6-hzxv6_32d29f1c-89cd-42f6-b9b6-e08b28e47266/router/0.log" Jul 10 22:06:07.227179 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:07.227090 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-546985876b-8jr5b_ea106490-e54e-4ba7-ab0a-7559bec3cb28/manager/0.log" Jul 10 22:06:07.496481 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:07.496388 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vt9b7_8df4b9d9-8f7f-4211-b328-e61375820b42/istio-proxy/0.log" Jul 10 22:06:07.703321 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:07.703275 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7mfz7_d6b9fa2c-57db-4859-bd54-72efa3c9a00d/discovery/0.log" Jul 10 22:06:07.896221 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:07.896188 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f44597c6-hzxv6_32d29f1c-89cd-42f6-b9b6-e08b28e47266/router/0.log" Jul 10 22:06:12.341394 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:12.341362 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-4hrsg_2e1cf7a2-c69d-46fb-bd7a-3d27aed69252/migrator/0.log" Jul 10 22:06:12.363164 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:12.363132 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-4hrsg_2e1cf7a2-c69d-46fb-bd7a-3d27aed69252/graceful-termination/0.log" Jul 10 22:06:12.734709 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:12.734675 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-5hgv9_9781f26e-f121-4bdf-a5df-6507501cf548/kube-storage-version-migrator-operator/1.log" Jul 10 22:06:12.736390 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:12.736366 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-5hgv9_9781f26e-f121-4bdf-a5df-6507501cf548/kube-storage-version-migrator-operator/0.log" Jul 10 22:06:13.835885 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.835858 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/kube-multus-additional-cni-plugins/0.log" Jul 10 22:06:13.857919 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.857878 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/egress-router-binary-copy/0.log" Jul 10 22:06:13.880168 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.880139 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/cni-plugins/0.log" Jul 10 22:06:13.904133 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.904108 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/bond-cni-plugin/0.log" Jul 10 22:06:13.928129 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.928104 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/routeoverride-cni/0.log" Jul 10 22:06:13.954168 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.954139 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/whereabouts-cni-bincopy/0.log" Jul 10 22:06:13.979469 ip-10-0-130-213 kubenswrapper[2561]: I0710 22:06:13.979441 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zczs_5014c522-e157-46e0-9fab-5df447a1d745/whereabouts-cni/0.log"