Sep 10 14:47:25.587154 ip-10-0-130-182 systemd[1]: Starting Kubernetes Kubelet... Sep 10 14:47:26.020231 ip-10-0-130-182 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:26.020231 ip-10-0-130-182 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 14:47:26.020231 ip-10-0-130-182 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:26.020231 ip-10-0-130-182 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 14:47:26.020231 ip-10-0-130-182 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:26.021317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.021224 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 14:47:26.026090 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026065 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.026090 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026089 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.026090 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026093 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026096 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026099 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026102 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026105 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026107 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026111 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026113 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026116 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026118 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026121 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026124 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026133 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026135 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026140 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026143 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026145 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026148 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026150 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.026195 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026153 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026156 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026158 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026161 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026163 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026166 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026168 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026171 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026174 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026176 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026179 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026182 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026185 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026190 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026194 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026198 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026201 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026205 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026207 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.026647 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026211 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026215 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026218 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026221 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026225 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026229 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026232 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026235 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026238 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026241 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026243 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026246 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026248 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026251 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026254 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026256 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026259 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026261 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026264 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026267 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.027135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026270 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026272 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026275 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026277 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026280 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026283 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026286 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026288 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026291 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026294 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026298 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026301 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026304 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026306 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026308 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026311 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026314 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026316 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026320 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026323 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.027635 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026326 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026328 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026331 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026333 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026336 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026338 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026341 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026343 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026347 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026350 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026354 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026357 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026360 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026362 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026365 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026368 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026370 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026373 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.026376 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027181 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.028135 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027189 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027192 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027197 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027201 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027204 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027207 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027210 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027213 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027217 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027220 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027223 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027226 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027228 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027232 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027235 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027238 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027241 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027243 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027246 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027248 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.028619 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027251 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027253 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027256 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027258 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027261 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027264 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027266 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027271 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027275 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027277 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027281 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027285 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027290 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027293 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027296 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027299 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027301 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027304 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027307 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.029160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027309 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027312 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027314 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027317 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027319 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027322 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027324 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027327 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027330 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027332 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027335 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027337 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027340 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027344 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027347 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027350 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027352 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027355 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027357 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027360 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.029667 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027362 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027365 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027368 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027370 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027373 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027376 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027379 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027381 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027384 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027386 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027389 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027391 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027394 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027396 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027399 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027401 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027404 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027406 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027409 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.030460 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027411 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027414 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027417 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027419 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027422 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027424 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027426 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027429 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027432 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027435 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027438 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027440 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027443 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027445 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027448 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027450 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027452 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027455 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027458 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027460 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.031050 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027891 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027897 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027900 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027903 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027906 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027909 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027912 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027914 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027919 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027922 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027925 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027928 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027932 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027935 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027937 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027940 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027942 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027945 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027948 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.031733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027953 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027956 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027960 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027963 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027966 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027969 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027972 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027974 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027977 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027980 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027982 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027985 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027987 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027990 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027993 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027995 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.027998 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028001 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028005 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028007 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.032276 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028010 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028012 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028014 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028017 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028020 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028022 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028025 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028028 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028031 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028033 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028036 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028038 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028042 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028045 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028047 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028051 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028053 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028056 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028058 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028061 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.032784 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028063 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028066 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028068 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028071 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028074 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028076 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028079 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028081 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028084 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028086 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028089 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028091 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028094 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028097 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028099 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028102 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028104 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028107 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028110 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.033296 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028112 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028115 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028118 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028120 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028125 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028130 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028133 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028135 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028139 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028142 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028144 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028147 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028150 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028153 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028155 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028158 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028160 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028163 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028166 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028168 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.033803 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.028171 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028253 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028290 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028909 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028915 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028920 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028923 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028927 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028932 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028935 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028939 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028942 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028946 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028950 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028955 2581 flags.go:64] FLAG: --cgroup-root="" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028958 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028961 2581 flags.go:64] FLAG: --client-ca-file="" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028965 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028968 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028972 2581 flags.go:64] FLAG: --cluster-domain="" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028975 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028978 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028982 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028985 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 14:47:26.034284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028989 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028992 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028995 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.028998 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029001 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029004 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029007 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029010 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029013 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029017 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029020 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029023 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029040 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029044 2581 flags.go:64] FLAG: --enable-server="true" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029047 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029052 2581 flags.go:64] FLAG: --event-burst="100" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029055 2581 flags.go:64] FLAG: --event-qps="50" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029058 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029061 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029064 2581 flags.go:64] FLAG: --eviction-hard="" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029068 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029071 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029074 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029077 2581 flags.go:64] FLAG: --eviction-soft="" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029080 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 14:47:26.034888 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029083 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029086 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029089 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029093 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029096 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029099 2581 flags.go:64] FLAG: --feature-gates="" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029103 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029106 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029109 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029112 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029115 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029119 2581 flags.go:64] FLAG: --help="false" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029121 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029125 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029128 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029130 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029134 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029137 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029140 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029143 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029145 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029148 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029151 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029154 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 14:47:26.035543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029157 2581 flags.go:64] FLAG: --kube-reserved="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029161 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029163 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029166 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029171 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029175 2581 flags.go:64] FLAG: --lock-file="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029178 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029182 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029185 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029190 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029193 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029196 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029199 2581 flags.go:64] FLAG: --logging-format="text" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029202 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029205 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029207 2581 flags.go:64] FLAG: --manifest-url="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029210 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029215 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029218 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029222 2581 flags.go:64] FLAG: --max-pods="110" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029225 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029228 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029231 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029234 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029237 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 14:47:26.036262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029240 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029243 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029251 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029254 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029257 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029260 2581 flags.go:64] FLAG: --pod-cidr="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029263 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029269 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029272 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029275 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029278 2581 flags.go:64] FLAG: --port="10250" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029283 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029285 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07939912201095571" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029289 2581 flags.go:64] FLAG: --qos-reserved="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029292 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029295 2581 flags.go:64] FLAG: --register-node="true" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029298 2581 flags.go:64] FLAG: --register-with-taints="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029302 2581 flags.go:64] FLAG: --registry-burst="10" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029305 2581 flags.go:64] FLAG: --registry-qps="5" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029308 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029311 2581 flags.go:64] FLAG: --reserved-memory="" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029315 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029318 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029321 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029324 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 14:47:26.036919 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029327 2581 flags.go:64] FLAG: --runonce="false" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029330 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029333 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029336 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029339 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029341 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029344 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029348 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029351 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029353 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029356 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029359 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029364 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029368 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029371 2581 flags.go:64] FLAG: --system-cgroups="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029373 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029379 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029382 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029387 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029390 2581 flags.go:64] FLAG: --tls-min-version="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029393 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029396 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029399 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029402 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029405 2581 flags.go:64] FLAG: --v="2" Sep 10 14:47:26.037565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029410 2581 flags.go:64] FLAG: --version="false" Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029414 2581 flags.go:64] FLAG: --vmodule="" Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029418 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.029421 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.029979 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030006 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030012 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030018 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030024 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030031 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030040 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030045 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030050 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030055 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030059 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030063 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030067 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030077 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030081 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030086 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.038291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030090 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030094 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030098 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030103 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030108 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030112 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030117 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030121 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030125 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030134 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030138 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030142 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030146 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030151 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030155 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030160 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030165 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030174 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030178 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030182 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.038809 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030186 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030190 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030198 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030203 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030208 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030212 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030216 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030221 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030225 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030229 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030234 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030239 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030243 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030247 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030256 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030261 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030265 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030270 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030274 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030279 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.039310 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030283 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030287 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030291 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030296 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030301 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030305 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030351 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030362 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030367 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030374 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030382 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030387 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030392 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030396 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030401 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030405 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030410 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030414 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030419 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.039822 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030423 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030432 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030437 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030441 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030445 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030451 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030455 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030460 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030464 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030468 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030473 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030477 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030482 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030493 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030498 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030510 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030514 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030522 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030527 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030532 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.040291 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030537 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030541 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030550 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.030557 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.031177 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.039711 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 14:47:26.040806 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.039881 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 14:47:26.041555 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.041543 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 14:47:26.041587 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.041557 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 14:47:26.041634 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041625 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.041634 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041634 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041639 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041642 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041645 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041648 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041651 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041653 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041657 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041661 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041666 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041668 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041671 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041674 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041677 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041680 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041682 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041685 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041688 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041691 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041694 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.041691 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041698 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041701 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041705 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041709 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041712 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041714 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041717 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041720 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041722 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041725 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041727 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041730 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041733 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041735 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041738 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041740 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041743 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041745 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041748 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.042267 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041750 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041780 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041783 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041786 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041788 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041791 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041794 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041796 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041799 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041802 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041804 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041806 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041809 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041812 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041814 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041818 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041821 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041823 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041826 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041829 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.042733 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041831 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041834 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041837 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041839 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041842 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041845 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041847 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041850 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041853 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041855 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041858 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041861 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041863 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041865 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041868 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041870 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041873 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041876 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041878 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041884 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.043231 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041886 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041889 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041891 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041894 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041896 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041899 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041901 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041904 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041906 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041909 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041912 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041914 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041917 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041920 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041922 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041925 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041929 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041932 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.043729 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.041935 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.041941 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042069 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042075 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042078 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042081 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042084 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042086 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042089 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042093 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042096 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042098 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042101 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042104 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042107 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:26.044181 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042110 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042112 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042115 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042117 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042120 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042123 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042125 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042128 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042131 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042133 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042137 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042140 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042143 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042145 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042148 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042150 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042153 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042156 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042159 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042162 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:26.044556 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042164 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042167 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042169 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042172 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042174 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042177 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042179 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042182 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042184 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042187 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042189 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042191 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042195 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042197 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042200 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042202 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042205 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042207 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042210 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042214 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:26.045066 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042219 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042222 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042224 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042227 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042230 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042233 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042236 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042239 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042241 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042243 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042247 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042250 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042252 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042255 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042257 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042260 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042262 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042264 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042267 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:26.045544 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042269 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042272 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042274 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042277 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042280 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042282 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042290 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042293 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042297 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042300 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042302 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042305 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042308 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042311 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042313 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042316 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042319 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042322 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042326 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:26.046033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042328 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042331 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042334 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042336 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042340 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042342 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042345 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:26.042347 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.042353 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.042498 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 14:47:26.046491 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.046015 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 14:47:26.047021 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.047008 2581 server.go:1019] "Starting client certificate rotation" Sep 10 14:47:26.047130 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.047112 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:47:26.047933 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.047922 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:47:26.051187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.051169 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:47:26.054119 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.054096 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:47:26.063564 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.063544 2581 log.go:25] "Validated CRI v1 runtime API" Sep 10 14:47:26.067265 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.067251 2581 log.go:25] "Validated CRI v1 image API" Sep 10 14:47:26.071458 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.071429 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 14:47:26.074080 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.074058 2581 fs.go:135] Filesystem UUIDs: map[24c3b6f8-a297-4391-890d-bca529d2c6f9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e58817fb-50fd-4dac-8f86-a4052bedb0a6:/dev/nvme0n1p3] Sep 10 14:47:26.074151 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.074080 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 14:47:26.080729 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.080580 2581 manager.go:216] Machine: {Timestamp:2026-09-10 14:47:26.079212273 +0000 UTC m=+0.373514393 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099351 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b23c8e4d79745922da3fec42e3ab SystemUUID:ec21b23c-8e4d-7974-5922-da3fec42e3ab BootID:2c0a0e4f-4a00-4d00-919a-8c6fa36665cd Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:f1:e4:48:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:f1:e4:48:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:32:8e:6e:48:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 14:47:26.080729 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.080703 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 14:47:26.080979 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.080871 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 14:47:26.084617 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.084596 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:47:26.084857 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.084819 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 14:47:26.085045 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.084862 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-182.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 14:47:26.085798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.085786 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 14:47:26.085831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.085800 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 14:47:26.085831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.085809 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:47:26.086595 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.086581 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:47:26.087804 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.087792 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 14:47:26.091929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.091912 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:47:26.092103 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092093 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 14:47:26.092134 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092128 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 10 14:47:26.092161 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092143 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 14:47:26.092187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092171 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 14:47:26.092187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092185 2581 kubelet.go:398] "Adding apiserver pod source" Sep 10 14:47:26.092240 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092197 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 14:47:26.092661 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092648 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:47:26.092698 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.092668 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:47:26.095348 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.095328 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 14:47:26.098637 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.098621 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 14:47:26.098696 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.098653 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 14:47:26.100287 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100274 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100291 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100298 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100304 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100314 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100319 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100333 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100338 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100346 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 14:47:26.100355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100352 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 14:47:26.100584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100361 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 14:47:26.100584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.100366 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 14:47:26.101903 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.101885 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 14:47:26.101903 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.101903 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 14:47:26.102147 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.102127 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 14:47:26.102194 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.102127 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 14:47:26.104364 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.104346 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-182.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 14:47:26.105905 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.105891 2581 server.go:1268] "Started kubelet" Sep 10 14:47:26.105982 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.105949 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 14:47:26.106108 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.106058 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 14:47:26.106161 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.106131 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 14:47:26.106834 ip-10-0-130-182 systemd[1]: Started Kubernetes Kubelet. Sep 10 14:47:26.107646 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.107561 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 14:47:26.108532 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.108513 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 10 14:47:26.111684 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.111662 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 14:47:26.114170 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.114147 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 14:47:26.114170 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.114158 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 14:47:26.114710 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.114334 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.115070 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115050 2581 factory.go:55] Registering systemd factory Sep 10 14:47:26.115070 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115070 2581 factory.go:223] Registration of the systemd container factory successfully Sep 10 14:47:26.115303 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115288 2581 factory.go:153] Registering CRI-O factory Sep 10 14:47:26.115303 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115308 2581 factory.go:223] Registration of the crio container factory successfully Sep 10 14:47:26.115409 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115360 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 14:47:26.115409 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115375 2581 factory.go:103] Registering Raw factory Sep 10 14:47:26.115409 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115397 2581 manager.go:1196] Started watching for new ooms in manager Sep 10 14:47:26.115850 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115782 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 14:47:26.115929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115812 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 14:47:26.115929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115888 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 14:47:26.116076 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.115993 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 10 14:47:26.116076 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.116001 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 10 14:47:26.116777 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.116741 2581 manager.go:349] Starting recovery of all containers Sep 10 14:47:26.120728 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.120704 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 14:47:26.120871 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.120852 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 14:47:26.122406 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.120561 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18d3fcfe4af7b7a9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-09-10 14:47:26.105860009 +0000 UTC m=+0.400162129,LastTimestamp:2026-09-10 14:47:26.105860009 +0000 UTC m=+0.400162129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Sep 10 14:47:26.127615 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.127574 2581 manager.go:354] Recovery completed Sep 10 14:47:26.129217 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.129177 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4ksf" Sep 10 14:47:26.132310 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.132294 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.134337 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.134315 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.134406 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.134354 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.134406 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.134366 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.134999 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.134982 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4ksf" Sep 10 14:47:26.135054 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.135013 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 14:47:26.135054 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.135029 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 14:47:26.135054 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.135050 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:47:26.135187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.135057 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.135990 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.135976 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.136038 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.136006 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.136038 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.136019 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.136330 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.136254 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18d3fcfe4caa3d2c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-09-10 14:47:26.134336812 +0000 UTC m=+0.428638940,LastTimestamp:2026-09-10 14:47:26.134336812 +0000 UTC m=+0.428638940,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Sep 10 14:47:26.137357 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.137342 2581 policy_none.go:49] "None policy: Start" Sep 10 14:47:26.137415 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.137360 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 14:47:26.137415 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.137371 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 14:47:26.138368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.138356 2581 policy_none.go:47] "Start" Sep 10 14:47:26.170491 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.170468 2581 manager.go:337] "Starting Device Plugin manager" Sep 10 14:47:26.170631 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.170513 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 14:47:26.170631 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.170526 2581 server.go:87] "Starting device plugin registration server" Sep 10 14:47:26.170884 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.170868 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 14:47:26.170932 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.170885 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 14:47:26.172933 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.171036 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 14:47:26.172933 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.171130 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 14:47:26.172933 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.171139 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 14:47:26.172933 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.172727 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 14:47:26.172933 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.172793 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.221696 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.221654 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 14:47:26.223012 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.222995 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 14:47:26.223113 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.223053 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 14:47:26.223113 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.223084 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 14:47:26.223205 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.223134 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 14:47:26.225579 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.225557 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:26.271536 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.271446 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.273100 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.273076 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.273183 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.273114 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.273183 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.273126 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.273183 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.273150 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.282102 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.282081 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.282160 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.282108 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-182.ec2.internal\": node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.309569 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.309532 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.324143 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.324092 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"] Sep 10 14:47:26.324294 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.324194 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.325314 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.325299 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.325389 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.325332 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.325389 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.325345 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.326625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.326611 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.326808 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.326795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.326849 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.326824 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.328599 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328579 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.328686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328616 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.328686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328627 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.328686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328579 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.328787 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328691 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.328787 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.328708 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.329918 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.329899 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.330013 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.329928 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:26.331382 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.331363 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:26.331467 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.331397 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:26.331467 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.331413 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:26.347442 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.347418 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.351724 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.351705 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.410601 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.410568 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.417572 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.417543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.417636 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.417575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.417636 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.417593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.510685 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.510644 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.518134 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.518232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.518232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.518321 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518231 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d36b70f3f303d4cb4f8a7bad8d41d64d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"d36b70f3f303d4cb4f8a7bad8d41d64d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.518321 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518237 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.518321 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.518247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/568a76fd77addf714f0c72a696f95cb0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"568a76fd77addf714f0c72a696f95cb0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.611646 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.611558 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.649133 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.649103 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.654938 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:26.654879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:26.712339 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.712300 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.812946 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.812911 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:26.913602 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:26.913517 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Sep 10 14:47:27.007130 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.007102 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:27.014346 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.014321 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Sep 10 14:47:27.026504 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.026469 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:47:27.028202 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.028182 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Sep 10 14:47:27.036675 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.036652 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:47:27.046840 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.046806 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 14:47:27.047023 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.047000 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 14:47:27.047023 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.047000 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 14:47:27.092847 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.092820 2581 apiserver.go:52] "Watching apiserver" Sep 10 14:47:27.100563 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.100534 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:27.104634 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.104609 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 14:47:27.105013 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.104992 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bkhbf","openshift-network-diagnostics/network-check-target-jxmnh","openshift-network-operator/iptables-alerter-2kz6r","openshift-ovn-kubernetes/ovnkube-node-zg967","kube-system/global-pull-secret-syncer-nhsnn","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2vz69","openshift-dns/node-resolver-9gr4p","openshift-image-registry/node-ca-dp6lf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","openshift-multus/multus-additional-cni-plugins-hvfqw","openshift-multus/multus-zsk8x","kube-system/konnectivity-agent-2xbnw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts"] Sep 10 14:47:27.106499 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.106479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.107701 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.107683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.108944 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.108922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.110068 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.110052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.110187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.110171 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.110261 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.110230 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hs8qf\"" Sep 10 14:47:27.110892 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.110877 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 14:47:27.110973 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.110919 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.111158 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111142 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 14:47:27.111249 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.111321 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.111303 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:27.111523 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mx8l5\"" Sep 10 14:47:27.111702 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111684 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 14:47:27.111784 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111685 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.111849 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.111831 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.114077 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.112496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.114077 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.113894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.114239 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.114225 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vv7f\"" Sep 10 14:47:27.114288 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.114255 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 14:47:27.114359 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.113896 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 14:47:27.114528 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.114510 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.114730 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.114710 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 14:47:27.114973 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.114955 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 14:47:27.115121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115093 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zsrsg\"" Sep 10 14:47:27.115180 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.115239 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115193 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.115239 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115194 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 14:47:27.115494 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115479 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 14:47:27.115565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115487 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.115624 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.115594 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qn8b9\"" Sep 10 14:47:27.116863 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.116842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.116959 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.116919 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:27.117340 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.117318 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.117937 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.117913 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 14:47:27.117937 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.117934 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.118085 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.117950 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fgjfj\"" Sep 10 14:47:27.118420 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.118393 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:27.118483 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.118469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:27.120005 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.119989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.121561 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-kubelet-config\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.121659 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-tmp\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.121659 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-netns\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121659 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvstk\" (UniqueName: \"kubernetes.io/projected/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kube-api-access-fvstk\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.121659 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-serviceca\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhmq\" (UniqueName: \"kubernetes.io/projected/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-kube-api-access-pnhmq\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-slash\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-var-lib-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-netd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121822 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.121859 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-device-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mnv\" (UniqueName: \"kubernetes.io/projected/238df29e-1e13-4956-9f42-ed8340fd572f-kube-api-access-98mnv\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-node-log\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.121983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-config\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-registration-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e57d46-a1e2-4060-b232-396e0ab2f28e-iptables-alerter-script\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e57d46-a1e2-4060-b232-396e0ab2f28e-host-slash\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-tuned\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122296 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-socket-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-host\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-modprobe-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-systemd-units\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-sys-fs\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wd7\" (UniqueName: \"kubernetes.io/projected/27e57d46-a1e2-4060-b232-396e0ab2f28e-kube-api-access-58wd7\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-sys\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-systemd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-ovn\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122609 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122618 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472jh\" (UniqueName: \"kubernetes.io/projected/3b52feef-393f-4999-a21d-6693bbdbcd7e-kube-api-access-472jh\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122635 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxl9b\"" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-run\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2qd\" (UniqueName: \"kubernetes.io/projected/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-kube-api-access-nw2qd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2582a9f4-efac-41a8-abb1-3ac3f0c04392-agent-certs\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.122868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-kubelet\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-etc-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2582a9f4-efac-41a8-abb1-3ac3f0c04392-konnectivity-ca\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-log-socket\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-script-lib\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-systemd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.122966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-host\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-lib-modules\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-dbus\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-bin\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-env-overrides\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysconfig\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-kubernetes\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-conf\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.123552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.123242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-var-lib-kubelet\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.124344 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124325 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 14:47:27.124344 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124336 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 14:47:27.124479 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124344 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 14:47:27.124479 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124368 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 14:47:27.124479 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124325 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 14:47:27.124479 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.124432 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-49sb6\"" Sep 10 14:47:27.125029 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.125015 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 14:47:27.125100 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.125088 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jz7xn\"" Sep 10 14:47:27.126352 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.126332 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:47:27.139706 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.139660 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:42:27 +0000 UTC" deadline="2028-06-05 10:27:39.514452574 +0000 UTC" Sep 10 14:47:27.139706 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.139702 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15211h40m12.374752858s" Sep 10 14:47:27.147221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.147193 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lszwv" Sep 10 14:47:27.155651 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.155622 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lszwv" Sep 10 14:47:27.217285 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.217249 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-dbus\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-bin\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-env-overrides\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysconfig\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-kubernetes\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-conf\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-var-lib-kubelet\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkbn\" (UniqueName: \"kubernetes.io/projected/7009b52c-fa14-4627-af75-b0af34939d26-kube-api-access-zkkbn\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-kubelet-config\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-tmp\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vlp\" (UniqueName: \"kubernetes.io/projected/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-kube-api-access-s7vlp\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224463 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224491 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-socket-dir-parent\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-conf-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.225014 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-netns\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-multus\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-daemon-config\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-multus-certs\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvstk\" (UniqueName: \"kubernetes.io/projected/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kube-api-access-fvstk\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-etc-kubernetes\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224952 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-serviceca\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.224981 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-bin\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-dbus\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.225050 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-kubernetes\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysctl-conf\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-sysconfig\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225299 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d109b72e-e301-4f56-9c25-296228e6257c-kubelet-config\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhmq\" (UniqueName: \"kubernetes.io/projected/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-kube-api-access-pnhmq\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225370 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-netns\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-slash\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-var-lib-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.225457 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:27.725428501 +0000 UTC m=+2.019730625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-var-lib-kubelet\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225707 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-var-lib-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225746 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225792 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-slash\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225927 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.225957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-netd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-cnibin\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226018 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-serviceca\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.226880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226037 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-cni-netd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-device-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mnv\" (UniqueName: \"kubernetes.io/projected/238df29e-1e13-4956-9f42-ed8340fd572f-kube-api-access-98mnv\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-device-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-node-log\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-system-cni-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-node-log\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-cnibin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-config\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-registration-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-registration-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e57d46-a1e2-4060-b232-396e0ab2f28e-iptables-alerter-script\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e57d46-a1e2-4060-b232-396e0ab2f28e-host-slash\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-tuned\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.227805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226626 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e57d46-a1e2-4060-b232-396e0ab2f28e-host-slash\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-socket-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-host\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.226713 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-tmp-dir\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226829 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-host\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-k8s-cni-cncf-io\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-socket-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.226874 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:27.726745077 +0000 UTC m=+2.021047184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mx7l\" (UniqueName: \"kubernetes.io/projected/d74bbe56-3432-43dd-9187-db6bcd3b4755-kube-api-access-5mx7l\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.226988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-modprobe-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227031 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-modprobe-d\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-system-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-hostroot\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-systemd-units\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.228584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-bin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-sys-fs\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wd7\" (UniqueName: \"kubernetes.io/projected/27e57d46-a1e2-4060-b232-396e0ab2f28e-kube-api-access-58wd7\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227803 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-sys\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-systemd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-ovn\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472jh\" (UniqueName: \"kubernetes.io/projected/3b52feef-393f-4999-a21d-6693bbdbcd7e-kube-api-access-472jh\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-kubelet\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-run\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2qd\" (UniqueName: \"kubernetes.io/projected/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-kube-api-access-nw2qd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2582a9f4-efac-41a8-abb1-3ac3f0c04392-agent-certs\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-kubelet\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.227999 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-config\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-etc-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-cni-binary-copy\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2582a9f4-efac-41a8-abb1-3ac3f0c04392-konnectivity-ca\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.229480 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-log-socket\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-script-lib\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-hosts-file\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-netns\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-env-overrides\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-systemd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-host\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-lib-modules\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228337 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27e57d46-a1e2-4060-b232-396e0ab2f28e-iptables-alerter-script\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228396 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-run\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-lib-modules\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228472 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-etc-selinux\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-systemd-units\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228553 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228593 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1b40d57-e4f8-4962-901d-2fd68dcdc287-sys-fs\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228746 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-ovn\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-sys\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230232 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.228880 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-run-systemd\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229134 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229696 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-systemd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229881 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2582a9f4-efac-41a8-abb1-3ac3f0c04392-konnectivity-ca\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229932 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-host\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-log-socket\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.229972 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-host-kubelet\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.230023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b52feef-393f-4999-a21d-6693bbdbcd7e-etc-openvswitch\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.230292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-tmp\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.230400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.230405 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-etc-tuned\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.230864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.230619 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b52feef-393f-4999-a21d-6693bbdbcd7e-ovnkube-script-lib\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.232446 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.232428 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2582a9f4-efac-41a8-abb1-3ac3f0c04392-agent-certs\") pod \"konnectivity-agent-2xbnw\" (UID: \"2582a9f4-efac-41a8-abb1-3ac3f0c04392\") " pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.234870 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.234843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvstk\" (UniqueName: \"kubernetes.io/projected/f1b40d57-e4f8-4962-901d-2fd68dcdc287-kube-api-access-fvstk\") pod \"aws-ebs-csi-driver-node-9rhts\" (UID: \"f1b40d57-e4f8-4962-901d-2fd68dcdc287\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.235293 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.235195 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhmq\" (UniqueName: \"kubernetes.io/projected/5d602dc7-5a71-455b-bc86-7f096b2cb9c8-kube-api-access-pnhmq\") pod \"node-ca-dp6lf\" (UID: \"5d602dc7-5a71-455b-bc86-7f096b2cb9c8\") " pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.235823 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.235797 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mnv\" (UniqueName: \"kubernetes.io/projected/238df29e-1e13-4956-9f42-ed8340fd572f-kube-api-access-98mnv\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.237536 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.237509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wd7\" (UniqueName: \"kubernetes.io/projected/27e57d46-a1e2-4060-b232-396e0ab2f28e-kube-api-access-58wd7\") pod \"iptables-alerter-2kz6r\" (UID: \"27e57d46-a1e2-4060-b232-396e0ab2f28e\") " pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.237904 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.237883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2qd\" (UniqueName: \"kubernetes.io/projected/0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b-kube-api-access-nw2qd\") pod \"tuned-2vz69\" (UID: \"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b\") " pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.240401 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.240369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-472jh\" (UniqueName: \"kubernetes.io/projected/3b52feef-393f-4999-a21d-6693bbdbcd7e-kube-api-access-472jh\") pod \"ovnkube-node-zg967\" (UID: \"3b52feef-393f-4999-a21d-6693bbdbcd7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.286633 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.286584 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36b70f3f303d4cb4f8a7bad8d41d64d.slice/crio-e520af2e8b6bd4f696f78289934afa130f9f25776eda303c30b1d095d7457db0 WatchSource:0}: Error finding container e520af2e8b6bd4f696f78289934afa130f9f25776eda303c30b1d095d7457db0: Status 404 returned error can't find the container with id e520af2e8b6bd4f696f78289934afa130f9f25776eda303c30b1d095d7457db0 Sep 10 14:47:27.292358 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.292336 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:47:27.300102 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.300073 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:27.329080 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkbn\" (UniqueName: \"kubernetes.io/projected/7009b52c-fa14-4627-af75-b0af34939d26-kube-api-access-zkkbn\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329080 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vlp\" (UniqueName: \"kubernetes.io/projected/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-kube-api-access-s7vlp\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-socket-dir-parent\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-conf-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-multus\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329276 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-conf-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-daemon-config\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329319 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-socket-dir-parent\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-multus-certs\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-multus\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-etc-kubernetes\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-multus-certs\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329425 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-etc-kubernetes\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-cnibin\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-system-cni-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329539 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-system-cni-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-cnibin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-cnibin\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-tmp-dir\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-k8s-cni-cncf-io\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mx7l\" (UniqueName: \"kubernetes.io/projected/d74bbe56-3432-43dd-9187-db6bcd3b4755-kube-api-access-5mx7l\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329674 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-cnibin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.329875 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-system-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-hostroot\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-bin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-kubelet\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-cni-binary-copy\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-hosts-file\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-netns\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329905 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-k8s-cni-cncf-io\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329911 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-multus-daemon-config\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329971 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-cni-bin\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.329995 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-tmp-dir\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-run-netns\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330087 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-system-cni-dir\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7009b52c-fa14-4627-af75-b0af34939d26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330151 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330142 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-host-var-lib-kubelet\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d74bbe56-3432-43dd-9187-db6bcd3b4755-hostroot\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.330673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330230 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.331273 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330260 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-hosts-file\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.331273 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7009b52c-fa14-4627-af75-b0af34939d26-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.331273 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.330508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d74bbe56-3432-43dd-9187-db6bcd3b4755-cni-binary-copy\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.336373 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.336350 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:27.336475 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.336381 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:27.336475 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.336396 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:27.336566 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.336477 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:27.836455846 +0000 UTC m=+2.130757979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:27.338558 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.338526 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkbn\" (UniqueName: \"kubernetes.io/projected/7009b52c-fa14-4627-af75-b0af34939d26-kube-api-access-zkkbn\") pod \"multus-additional-cni-plugins-hvfqw\" (UID: \"7009b52c-fa14-4627-af75-b0af34939d26\") " pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.338663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.338634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vlp\" (UniqueName: \"kubernetes.io/projected/33eedfe2-81ba-4f3a-ba9c-5206a372afd8-kube-api-access-s7vlp\") pod \"node-resolver-9gr4p\" (UID: \"33eedfe2-81ba-4f3a-ba9c-5206a372afd8\") " pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.338663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.338645 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mx7l\" (UniqueName: \"kubernetes.io/projected/d74bbe56-3432-43dd-9187-db6bcd3b4755-kube-api-access-5mx7l\") pod \"multus-zsk8x\" (UID: \"d74bbe56-3432-43dd-9187-db6bcd3b4755\") " pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.350706 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.350680 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568a76fd77addf714f0c72a696f95cb0.slice/crio-3d713476eee9b0c8a88d4226bba92f86c7fad36dc9729d5e82be45deac881478 WatchSource:0}: Error finding container 3d713476eee9b0c8a88d4226bba92f86c7fad36dc9729d5e82be45deac881478: Status 404 returned error can't find the container with id 3d713476eee9b0c8a88d4226bba92f86c7fad36dc9729d5e82be45deac881478 Sep 10 14:47:27.434188 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.434082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2vz69" Sep 10 14:47:27.440812 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.440788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:27.443033 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.443006 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3a1449_5fb6_4e4f_b4a4_2a85e7d77e3b.slice/crio-fd947f51d3adf5aef714173adf201221b455c9ff8a8ae3477c272907f188e9d7 WatchSource:0}: Error finding container fd947f51d3adf5aef714173adf201221b455c9ff8a8ae3477c272907f188e9d7: Status 404 returned error can't find the container with id fd947f51d3adf5aef714173adf201221b455c9ff8a8ae3477c272907f188e9d7 Sep 10 14:47:27.449555 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.449529 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2582a9f4_efac_41a8_abb1_3ac3f0c04392.slice/crio-2351c365c8fa8c56187f8b4161bec936a30b9f7a6a784b84148a77e3902e74f7 WatchSource:0}: Error finding container 2351c365c8fa8c56187f8b4161bec936a30b9f7a6a784b84148a77e3902e74f7: Status 404 returned error can't find the container with id 2351c365c8fa8c56187f8b4161bec936a30b9f7a6a784b84148a77e3902e74f7 Sep 10 14:47:27.469728 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.469672 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kz6r" Sep 10 14:47:27.478620 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.478585 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e57d46_a1e2_4060_b232_396e0ab2f28e.slice/crio-ef9017b96e39439d247fb5d93027ce60d4af8d81defcf4de42fcfbef0747a245 WatchSource:0}: Error finding container ef9017b96e39439d247fb5d93027ce60d4af8d81defcf4de42fcfbef0747a245: Status 404 returned error can't find the container with id ef9017b96e39439d247fb5d93027ce60d4af8d81defcf4de42fcfbef0747a245 Sep 10 14:47:27.482816 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.482794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:27.491029 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.490991 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b52feef_393f_4999_a21d_6693bbdbcd7e.slice/crio-902be9e5febf98c1353d158cacaa6f46ebc852315a2e5c964b5c333dff188c05 WatchSource:0}: Error finding container 902be9e5febf98c1353d158cacaa6f46ebc852315a2e5c964b5c333dff188c05: Status 404 returned error can't find the container with id 902be9e5febf98c1353d158cacaa6f46ebc852315a2e5c964b5c333dff188c05 Sep 10 14:47:27.499191 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.499170 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" Sep 10 14:47:27.507226 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.507192 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b40d57_e4f8_4962_901d_2fd68dcdc287.slice/crio-12739868ad6992aec221fb83facca1ca462f1f069b908e7723259a4ccb0e404c WatchSource:0}: Error finding container 12739868ad6992aec221fb83facca1ca462f1f069b908e7723259a4ccb0e404c: Status 404 returned error can't find the container with id 12739868ad6992aec221fb83facca1ca462f1f069b908e7723259a4ccb0e404c Sep 10 14:47:27.513810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.513788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dp6lf" Sep 10 14:47:27.522948 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.522906 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d602dc7_5a71_455b_bc86_7f096b2cb9c8.slice/crio-f513d61f359af6e9b34f17eae95973b5a308852ec3ca126ed54138c74ce5b0df WatchSource:0}: Error finding container f513d61f359af6e9b34f17eae95973b5a308852ec3ca126ed54138c74ce5b0df: Status 404 returned error can't find the container with id f513d61f359af6e9b34f17eae95973b5a308852ec3ca126ed54138c74ce5b0df Sep 10 14:47:27.529400 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.529375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gr4p" Sep 10 14:47:27.539177 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.539148 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eedfe2_81ba_4f3a_ba9c_5206a372afd8.slice/crio-ef1ead84734adcbf9d7f83d5e9e780723b0194d43645d28712e5bca25bce021d WatchSource:0}: Error finding container ef1ead84734adcbf9d7f83d5e9e780723b0194d43645d28712e5bca25bce021d: Status 404 returned error can't find the container with id ef1ead84734adcbf9d7f83d5e9e780723b0194d43645d28712e5bca25bce021d Sep 10 14:47:27.558578 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.558541 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" Sep 10 14:47:27.563336 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.563308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zsk8x" Sep 10 14:47:27.569744 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.569709 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009b52c_fa14_4627_af75_b0af34939d26.slice/crio-deeb86f47a6274d7753be11036db60b57e7a4383f14593c7253715201af705f9 WatchSource:0}: Error finding container deeb86f47a6274d7753be11036db60b57e7a4383f14593c7253715201af705f9: Status 404 returned error can't find the container with id deeb86f47a6274d7753be11036db60b57e7a4383f14593c7253715201af705f9 Sep 10 14:47:27.573262 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:47:27.573231 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74bbe56_3432_43dd_9187_db6bcd3b4755.slice/crio-50d65137b3eb9e64f0374ab0e69e1157072ee98547f66ed722bbbd894e4f5037 WatchSource:0}: Error finding container 50d65137b3eb9e64f0374ab0e69e1157072ee98547f66ed722bbbd894e4f5037: Status 404 returned error can't find the container with id 50d65137b3eb9e64f0374ab0e69e1157072ee98547f66ed722bbbd894e4f5037 Sep 10 14:47:27.732301 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.732191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:27.732301 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.732253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:27.732535 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.732343 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:27.732535 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.732370 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:27.732535 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.732404 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:28.732388999 +0000 UTC m=+3.026691107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:27.732535 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.732434 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:28.732415845 +0000 UTC m=+3.026717973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:27.933077 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:27.933038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:27.933250 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.933167 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:27.933250 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.933187 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:27.933250 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.933197 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:27.933250 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:27.933250 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:28.933233601 +0000 UTC m=+3.227535709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:28.157543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.157403 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:28 +0000 UTC" deadline="2028-02-19 13:19:44.709654178 +0000 UTC" Sep 10 14:47:28.157543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.157481 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12646h32m16.552177232s" Sep 10 14:47:28.226764 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.226725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:28.226972 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.226897 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:28.227356 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.227321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:28.227463 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.227422 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:28.241670 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.241601 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"d36b70f3f303d4cb4f8a7bad8d41d64d","Type":"ContainerStarted","Data":"e520af2e8b6bd4f696f78289934afa130f9f25776eda303c30b1d095d7457db0"} Sep 10 14:47:28.251312 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.251265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerStarted","Data":"deeb86f47a6274d7753be11036db60b57e7a4383f14593c7253715201af705f9"} Sep 10 14:47:28.255138 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.255099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gr4p" event={"ID":"33eedfe2-81ba-4f3a-ba9c-5206a372afd8","Type":"ContainerStarted","Data":"ef1ead84734adcbf9d7f83d5e9e780723b0194d43645d28712e5bca25bce021d"} Sep 10 14:47:28.257314 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.257283 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" event={"ID":"f1b40d57-e4f8-4962-901d-2fd68dcdc287","Type":"ContainerStarted","Data":"12739868ad6992aec221fb83facca1ca462f1f069b908e7723259a4ccb0e404c"} Sep 10 14:47:28.259679 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.259636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2xbnw" event={"ID":"2582a9f4-efac-41a8-abb1-3ac3f0c04392","Type":"ContainerStarted","Data":"2351c365c8fa8c56187f8b4161bec936a30b9f7a6a784b84148a77e3902e74f7"} Sep 10 14:47:28.265379 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.265110 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zsk8x" event={"ID":"d74bbe56-3432-43dd-9187-db6bcd3b4755","Type":"ContainerStarted","Data":"50d65137b3eb9e64f0374ab0e69e1157072ee98547f66ed722bbbd894e4f5037"} Sep 10 14:47:28.267238 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.267186 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dp6lf" event={"ID":"5d602dc7-5a71-455b-bc86-7f096b2cb9c8","Type":"ContainerStarted","Data":"f513d61f359af6e9b34f17eae95973b5a308852ec3ca126ed54138c74ce5b0df"} Sep 10 14:47:28.270098 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.269322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"902be9e5febf98c1353d158cacaa6f46ebc852315a2e5c964b5c333dff188c05"} Sep 10 14:47:28.271286 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.271240 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kz6r" event={"ID":"27e57d46-a1e2-4060-b232-396e0ab2f28e","Type":"ContainerStarted","Data":"ef9017b96e39439d247fb5d93027ce60d4af8d81defcf4de42fcfbef0747a245"} Sep 10 14:47:28.276176 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.276127 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2vz69" event={"ID":"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b","Type":"ContainerStarted","Data":"fd947f51d3adf5aef714173adf201221b455c9ff8a8ae3477c272907f188e9d7"} Sep 10 14:47:28.278282 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.278244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerStarted","Data":"3d713476eee9b0c8a88d4226bba92f86c7fad36dc9729d5e82be45deac881478"} Sep 10 14:47:28.623467 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.623377 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:28.741448 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.741329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:28.741659 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.741488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:28.741730 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.741681 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:28.741815 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.741748 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:30.741729264 +0000 UTC m=+5.036031376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:28.742301 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.742275 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:28.742388 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.742342 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:30.742324865 +0000 UTC m=+5.036626979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:28.943135 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:28.943036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:28.943431 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.943310 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:28.943431 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.943338 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:28.943431 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.943351 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:28.943431 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:28.943414 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:30.943395482 +0000 UTC m=+5.237697619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:29.158182 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:29.158144 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:28 +0000 UTC" deadline="2028-02-22 16:17:54.022416361 +0000 UTC" Sep 10 14:47:29.158182 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:29.158177 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12721h30m24.864242778s" Sep 10 14:47:29.223910 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:29.223752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:29.224085 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:29.223986 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:30.226996 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.226256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:30.226996 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.226403 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:30.228370 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.228130 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:30.228370 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.228274 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:30.237869 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.237612 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.760906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.761000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.761137 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.761210 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:34.76118747 +0000 UTC m=+9.055489584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.761818 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:30.761961 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.761900 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:34.761882226 +0000 UTC m=+9.056184343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:30.962462 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:30.962414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:30.962667 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.962647 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:30.962738 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.962672 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:30.962738 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.962686 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:30.962879 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:30.962751 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:34.962731727 +0000 UTC m=+9.257033839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:31.224940 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:31.224327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:31.224940 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:31.224475 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:32.225785 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:32.224948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:32.225785 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:32.225074 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:32.225785 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:32.225077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:32.225785 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:32.225141 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:33.223972 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:33.223927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:33.224165 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:33.224080 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:34.224310 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:34.224268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:34.224810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:34.224322 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:34.224810 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.224432 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:34.224959 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.224834 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:34.796177 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:34.796125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:34.796385 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:34.796224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:34.796385 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.796370 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:34.796510 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.796437 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:42.796415811 +0000 UTC m=+17.090717926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:34.797014 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.796988 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:34.797126 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.797051 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:42.79703516 +0000 UTC m=+17.091337269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:34.998032 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:34.997930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:34.998225 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.998129 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:34.998225 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.998158 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:34.998225 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.998173 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:34.998402 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:34.998249 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:42.998228851 +0000 UTC m=+17.292530966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:35.224864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:35.224297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:35.224864 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:35.224469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:36.224648 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:36.224611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:36.224818 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:36.224704 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:36.224908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:36.224882 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:36.225279 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:36.225009 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:37.224246 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:37.224196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:37.224440 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:37.224357 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:38.224854 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:38.224277 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:38.224854 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:38.224294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:38.224854 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:38.224416 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:38.224854 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:38.224463 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:39.224218 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:39.224174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:39.224382 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:39.224304 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:40.224049 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:40.224006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:40.224482 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:40.224138 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:40.224482 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:40.224212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:40.224482 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:40.224303 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:41.223820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:41.223710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:41.223986 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:41.223854 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:42.224184 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:42.224148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:42.224819 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.224289 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:42.224819 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:42.224148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:42.224819 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.224742 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:42.866021 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:42.865980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:42.866233 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:42.866061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:42.866233 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.866205 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:42.866338 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.866271 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:47:58.86625135 +0000 UTC m=+33.160553479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:42.866664 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.866643 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:42.866792 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:42.866702 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:47:58.866685235 +0000 UTC m=+33.160987366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:43.068003 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:43.067966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:43.068178 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:43.068117 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:43.068178 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:43.068136 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:43.068178 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:43.068148 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:43.068355 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:43.068211 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:59.068194214 +0000 UTC m=+33.362496340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:43.223615 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:43.223525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:43.223819 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:43.223677 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:44.223741 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:44.223261 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:44.223741 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:44.223271 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:44.223741 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:44.223392 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:44.223741 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:44.223516 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:45.223507 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:45.223471 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:45.223716 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:45.223619 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:46.224597 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:46.224153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:46.224597 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:46.224276 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:46.224597 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:46.224338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:46.224597 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:46.224463 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:47.223624 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:47.223591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:47.223828 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:47.223787 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:48.227287 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:48.223913 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:48.227287 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:48.224034 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:48.227287 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:48.224487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:48.227287 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:48.224578 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:49.223658 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.223314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:49.223873 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:49.223827 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:49.342136 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.342104 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="0b1c5b3ad47080ea98bbde81bf3bac60733ba46c5206684abff18f524933134c" exitCode=0 Sep 10 14:47:49.342963 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.342196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"0b1c5b3ad47080ea98bbde81bf3bac60733ba46c5206684abff18f524933134c"} Sep 10 14:47:49.343825 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.343802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gr4p" event={"ID":"33eedfe2-81ba-4f3a-ba9c-5206a372afd8","Type":"ContainerStarted","Data":"83c588c1e99108e250662534640ec83caa48950b31f4764e65aa25f7319f8bd0"} Sep 10 14:47:49.345420 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.345401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" event={"ID":"f1b40d57-e4f8-4962-901d-2fd68dcdc287","Type":"ContainerStarted","Data":"63ff608a92409ab4b426431ba3c4c693772e3c58ce011e7c43e1ee504c5ffb11"} Sep 10 14:47:49.346904 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.346878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2xbnw" event={"ID":"2582a9f4-efac-41a8-abb1-3ac3f0c04392","Type":"ContainerStarted","Data":"699cd357543b0385500035c36c610ea9de77898ef6a4402c2dfd70f1f859e251"} Sep 10 14:47:49.348394 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.348373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zsk8x" event={"ID":"d74bbe56-3432-43dd-9187-db6bcd3b4755","Type":"ContainerStarted","Data":"9eadc4a21221df471c4aac3e069670e6ded94296b3a6837675d672dc784e5596"} Sep 10 14:47:49.349790 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.349749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dp6lf" event={"ID":"5d602dc7-5a71-455b-bc86-7f096b2cb9c8","Type":"ContainerStarted","Data":"ebbfe36e4ae684a2fa38781f91e38482c3946664accbbe59b4c6682df2964cc1"} Sep 10 14:47:49.352711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352690 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"7afcc39632856613cb1346730b80ab1e3e74fd79cd29764e0ecf69f6ee959d6e"} Sep 10 14:47:49.352823 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"3baba2ebea3737b4070f1e2be1ece41ee7cdc7cb52e83a7fc103b5aac6e6de03"} Sep 10 14:47:49.352823 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"c7eb0e647d68928651617f5688b14e208ba14f83bf4ce19fe023756a27531a80"} Sep 10 14:47:49.352897 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"1c4eca3a2234b7ecddcebb28e779f3078ee7d8ca03c07b736a8a5d6ce1b7b71c"} Sep 10 14:47:49.352897 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"f4b0632032df5b8f94887c0855f7289ed2deaf05e9d1edfd3fa1fafcd4ec8de3"} Sep 10 14:47:49.352967 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.352911 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"55b9729f9324f911fbb2e0ed1c9665df99ab6218a79af84630595eacb064ec76"} Sep 10 14:47:49.354162 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.354141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2vz69" event={"ID":"0a3a1449-5fb6-4e4f-b4a4-2a85e7d77e3b","Type":"ContainerStarted","Data":"d4b88bf678437b1ed1feb9fb49612a32642b07c75792295055de4231df0aa7c1"} Sep 10 14:47:49.355690 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.355669 2581 generic.go:356] "Generic (PLEG): container finished" podID="568a76fd77addf714f0c72a696f95cb0" containerID="268474a089cc59bc69a5946d4dd05b85293f556c768640120509785c9b48fc93" exitCode=0 Sep 10 14:47:49.355799 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.355735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerDied","Data":"268474a089cc59bc69a5946d4dd05b85293f556c768640120509785c9b48fc93"} Sep 10 14:47:49.357240 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.357218 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"d36b70f3f303d4cb4f8a7bad8d41d64d","Type":"ContainerStarted","Data":"5ec56b3b26a959954fbefcbadfb92d295a66d1cc1ca13b88c209f6f797c3b9fe"} Sep 10 14:47:49.377209 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.377125 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zsk8x" podStartSLOduration=1.6710685220000001 podStartE2EDuration="22.377104637s" totalImagesPullingTime="20.706036115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.376258257 +0000 UTC m=+23.670560387" watchObservedRunningTime="2026-09-10 14:47:49.377104637 +0000 UTC m=+23.671406767" Sep 10 14:47:49.397090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.397032 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2vz69" podStartSLOduration=1.6179534530000002 podStartE2EDuration="22.39701469s" totalImagesPullingTime="20.779061237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.396958974 +0000 UTC m=+23.691261102" watchObservedRunningTime="2026-09-10 14:47:49.39701469 +0000 UTC m=+23.691316818" Sep 10 14:47:49.412250 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.412184 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2xbnw" podStartSLOduration=1.642634559 podStartE2EDuration="22.412165081s" totalImagesPullingTime="20.769530522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.411209409 +0000 UTC m=+23.705511539" watchObservedRunningTime="2026-09-10 14:47:49.412165081 +0000 UTC m=+23.706467211" Sep 10 14:47:49.435404 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.435347 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9gr4p" podStartSLOduration=1.753823083 podStartE2EDuration="22.435329806s" totalImagesPullingTime="20.681506723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.434735192 +0000 UTC m=+23.729037322" watchObservedRunningTime="2026-09-10 14:47:49.435329806 +0000 UTC m=+23.729631934" Sep 10 14:47:49.470929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.470851 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" podStartSLOduration=21.470829707 podStartE2EDuration="21.470829707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.469955233 +0000 UTC m=+23.764257364" watchObservedRunningTime="2026-09-10 14:47:49.470829707 +0000 UTC m=+23.765131837" Sep 10 14:47:49.489455 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:49.489396 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dp6lf" podStartSLOduration=1.794934587 podStartE2EDuration="22.489371493s" totalImagesPullingTime="20.694436906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.488731951 +0000 UTC m=+23.783034086" watchObservedRunningTime="2026-09-10 14:47:49.489371493 +0000 UTC m=+23.783673623" Sep 10 14:47:50.223862 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.223655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:50.224008 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.223689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:50.224008 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:50.223949 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:50.224121 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:50.224033 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:50.232223 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.232194 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 14:47:50.362405 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.362257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"568a76fd77addf714f0c72a696f95cb0","Type":"ContainerStarted","Data":"718b0562ff2ba136497de98e55a561c696808fcbc0864545778db7d1d853ee74"} Sep 10 14:47:50.364657 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.364617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" event={"ID":"f1b40d57-e4f8-4962-901d-2fd68dcdc287","Type":"ContainerStarted","Data":"a358e2557680357a81e90c2a976ff0e75b575ef0524ddbf87e76a9190e79d639"} Sep 10 14:47:50.366583 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.366549 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kz6r" event={"ID":"27e57d46-a1e2-4060-b232-396e0ab2f28e","Type":"ContainerStarted","Data":"493456eb12067035ca18e5529fbd625ed2307d819c415bac83989e73d2ca6334"} Sep 10 14:47:50.377285 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.377224 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" podStartSLOduration=22.377202032 podStartE2EDuration="22.377202032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:50.377054193 +0000 UTC m=+24.671356321" watchObservedRunningTime="2026-09-10 14:47:50.377202032 +0000 UTC m=+24.671504163" Sep 10 14:47:50.390464 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:50.390413 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2kz6r" podStartSLOduration=2.652622006 podStartE2EDuration="23.390395991s" totalImagesPullingTime="20.737773985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:50.390098531 +0000 UTC m=+24.684400661" watchObservedRunningTime="2026-09-10 14:47:50.390395991 +0000 UTC m=+24.684698151" Sep 10 14:47:51.187115 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:51.187017 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:47:50.232216992Z","UUID":"4e2b1169-0629-44b3-b01f-5082511ab8da","Handler":null,"Name":"","Endpoint":""} Sep 10 14:47:51.188712 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:51.188688 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 14:47:51.188874 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:51.188728 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 14:47:51.223513 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:51.223476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:51.223716 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:51.223620 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:51.379484 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:51.379386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"039a502f9a7db20f940f3b073a81266e91d979c5e7b496713320583301b44275"} Sep 10 14:47:52.224344 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:52.224246 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:52.224517 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:52.224361 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:52.224517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:52.224411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:52.224517 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:52.224505 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:52.383167 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:52.383128 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" event={"ID":"f1b40d57-e4f8-4962-901d-2fd68dcdc287","Type":"ContainerStarted","Data":"5a7ffb48b2ff6bbd42fdb90f55e3ca258b56413230aca52af24b9fa49f9d251e"} Sep 10 14:47:52.400586 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:52.400537 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9rhts" podStartSLOduration=5.3836870900000005 podStartE2EDuration="25.400520967s" totalImagesPullingTime="20.016833877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:52.399409066 +0000 UTC m=+26.693711196" watchObservedRunningTime="2026-09-10 14:47:52.400520967 +0000 UTC m=+26.694823095" Sep 10 14:47:53.223977 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:53.223944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:53.224143 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:53.224060 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:53.785925 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:53.785888 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:53.786687 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:53.786657 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:54.224138 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:54.224064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:54.224284 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:54.224064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:54.224284 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:54.224211 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:54.224284 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:54.224259 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:54.388477 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:54.388434 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:54.389131 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:54.389110 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2xbnw" Sep 10 14:47:55.223737 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.223493 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:55.224385 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:55.223867 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:55.395189 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.395148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" event={"ID":"3b52feef-393f-4999-a21d-6693bbdbcd7e","Type":"ContainerStarted","Data":"09fbc3718f0aa8511d9f1de891b22a06d4529d2a4d5d6c33f0d1804ce86299bd"} Sep 10 14:47:55.395540 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.395398 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:55.395540 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.395467 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:55.397235 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.397201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerStarted","Data":"520afc801fe01fdb531308876fc26e6f9016689a9fd8b9dcc98da444f93ea0f1"} Sep 10 14:47:55.411800 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.411712 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:55.422125 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:55.422076 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" podStartSLOduration=7.6362326750000005 podStartE2EDuration="28.42205942s" totalImagesPullingTime="20.785826745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:55.42158824 +0000 UTC m=+29.715890368" watchObservedRunningTime="2026-09-10 14:47:55.42205942 +0000 UTC m=+29.716361550" Sep 10 14:47:56.228249 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.228220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:56.228786 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:56.228341 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:56.228786 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.228444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:56.228786 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:56.228556 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:56.401135 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.401101 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="520afc801fe01fdb531308876fc26e6f9016689a9fd8b9dcc98da444f93ea0f1" exitCode=0 Sep 10 14:47:56.401333 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.401197 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"520afc801fe01fdb531308876fc26e6f9016689a9fd8b9dcc98da444f93ea0f1"} Sep 10 14:47:56.401805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.401779 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:56.418858 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.418828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:47:56.986484 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.986451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhsnn"] Sep 10 14:47:56.986654 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.986587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:56.986725 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:56.986700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.989940 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkhbf"] Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.990096 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:56.990237 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.990304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxmnh"] Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:56.990376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:56.990517 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:56.990486 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:47:57.406747 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:57.406712 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="09590d97eab01e2ea4a8582ece1859468f2fa51323a231236cd5e5e78ed90fca" exitCode=0 Sep 10 14:47:57.407355 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:57.406798 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"09590d97eab01e2ea4a8582ece1859468f2fa51323a231236cd5e5e78ed90fca"} Sep 10 14:47:58.223722 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.223506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:58.223889 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.223591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:58.223889 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.223857 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:47:58.223976 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.223924 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:47:58.411152 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.411107 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="ee17cf1a5f8dd64f6ce15d084886255963101156a3fadd63cd99735e05fd0da4" exitCode=0 Sep 10 14:47:58.411522 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.411199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"ee17cf1a5f8dd64f6ce15d084886255963101156a3fadd63cd99735e05fd0da4"} Sep 10 14:47:58.887227 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.887188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:47:58.887418 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:58.887254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:47:58.887418 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.887359 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:58.887515 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.887429 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret podName:d109b72e-e301-4f56-9c25-296228e6257c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:30.887413575 +0000 UTC m=+65.181715681 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret") pod "global-pull-secret-syncer-nhsnn" (UID: "d109b72e-e301-4f56-9c25-296228e6257c") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:58.887515 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.887360 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:58.887515 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:58.887493 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:30.887474547 +0000 UTC m=+65.181776672 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:59.088356 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:59.088305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:59.088515 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:59.088493 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:59.088515 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:59.088516 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:59.088584 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:59.088526 2581 projected.go:196] Error preparing data for projected volume kube-api-access-k2j9x for pod openshift-network-diagnostics/network-check-target-jxmnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:59.088620 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:59.088583 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x podName:a25b18d9-742c-4589-bdd0-c90899b64ad9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:31.088566245 +0000 UTC m=+65.382868372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k2j9x" (UniqueName: "kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x") pod "network-check-target-jxmnh" (UID: "a25b18d9-742c-4589-bdd0-c90899b64ad9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:59.224015 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:47:59.223926 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:47:59.224160 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:47:59.224026 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxmnh" podUID="a25b18d9-742c-4589-bdd0-c90899b64ad9" Sep 10 14:48:00.223741 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:00.223662 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:48:00.224189 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:00.223823 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhsnn" podUID="d109b72e-e301-4f56-9c25-296228e6257c" Sep 10 14:48:00.224189 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:00.223832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:48:00.224189 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:00.223939 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bkhbf" podUID="238df29e-1e13-4956-9f42-ed8340fd572f" Sep 10 14:48:01.024246 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.024211 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeReady" Sep 10 14:48:01.024426 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.024391 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 14:48:01.058718 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.058631 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp"] Sep 10 14:48:01.069143 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.069107 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8"] Sep 10 14:48:01.069679 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.069657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" Sep 10 14:48:01.072394 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.072369 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.072540 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.072368 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.072540 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.072440 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5nsfj\"" Sep 10 14:48:01.073066 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.073040 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p"] Sep 10 14:48:01.073182 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.073158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.076219 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.076177 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xpx2r"] Sep 10 14:48:01.077257 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.076825 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 14:48:01.077257 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.076894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.077409 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.077267 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.076835 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v7wcd\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.079206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.081639 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.081687 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.081966 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vmpdh\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.082046 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.082612 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.084433 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj"] Sep 10 14:48:01.085106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.084564 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.087519 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.087497 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.087685 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.087556 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lmnd4\"" Sep 10 14:48:01.087685 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.087642 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 14:48:01.087969 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.087947 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.088259 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.088245 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 14:48:01.089129 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.089011 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:48:01.089221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.089197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.092939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.092916 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.093934 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.093911 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 14:48:01.094194 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.094171 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.094405 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.094389 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tpmrn\"" Sep 10 14:48:01.094565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.094550 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 14:48:01.096043 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.096015 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 14:48:01.098817 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.098038 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp"] Sep 10 14:48:01.098817 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.098078 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-chv6f"] Sep 10 14:48:01.098817 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.098175 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.101048 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.101025 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgzsd\"" Sep 10 14:48:01.101292 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.101271 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 14:48:01.101664 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.101524 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 14:48:01.101664 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.101639 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103109 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p"] Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103142 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8"] Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xpx2r"] Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj"] Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103213 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmd2s"] Sep 10 14:48:01.103788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.103788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" Sep 10 14:48:01.106471 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.106453 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hxzm2\"" Sep 10 14:48:01.106748 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.106710 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.106884 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.106871 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.107026 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.107006 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 14:48:01.108452 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.108433 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:48:01.108521 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.108464 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hr8bk"] Sep 10 14:48:01.109053 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.109034 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.115115 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.115054 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 14:48:01.115458 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.115437 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7c4td\"" Sep 10 14:48:01.116735 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.115657 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 14:48:01.117029 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.117006 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-chv6f"] Sep 10 14:48:01.117117 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.117053 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmd2s"] Sep 10 14:48:01.117117 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.117088 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr8bk"] Sep 10 14:48:01.117200 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.117174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.119540 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.119520 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 14:48:01.119661 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.119550 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 14:48:01.120119 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.120104 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5d77\"" Sep 10 14:48:01.120205 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.120163 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 14:48:01.202950 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.202915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4wk7\" (UniqueName: \"kubernetes.io/projected/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-kube-api-access-q4wk7\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.202950 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.202957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.203206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.202978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbb4t\" (UniqueName: \"kubernetes.io/projected/e5a0946b-0765-4156-b0f8-6759a19ab380-kube-api-access-tbb4t\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.203206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6af49c5-244d-47de-bb7f-3d6cfde75f04-cert\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.203206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-config-volume\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.203206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrqw\" (UniqueName: \"kubernetes.io/projected/d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32-kube-api-access-pdrqw\") pod \"volume-data-source-validator-695c895bd6-2tvmp\" (UID: \"d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" Sep 10 14:48:01.203206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4mw\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxpr\" (UniqueName: \"kubernetes.io/projected/b7d8884e-96c9-435b-8150-175a79139143-kube-api-access-5kxpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2sf2\" (UniqueName: \"kubernetes.io/projected/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-kube-api-access-l2sf2\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d8884e-96c9-435b-8150-175a79139143-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-config\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-serving-cert\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203372 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a0946b-0765-4156-b0f8-6759a19ab380-config\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203395 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d8884e-96c9-435b-8150-175a79139143-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tt5n\" (UniqueName: \"kubernetes.io/projected/98ce95dc-ee92-4834-92c4-212c5f260ba9-kube-api-access-6tt5n\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a0946b-0765-4156-b0f8-6759a19ab380-serving-cert\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndt4q\" (UniqueName: \"kubernetes.io/projected/8f9b3eb4-5d07-4088-b242-d23a709660ce-kube-api-access-ndt4q\") pod \"network-check-source-685456655f-chv6f\" (UID: \"8f9b3eb4-5d07-4088-b242-d23a709660ce\") " pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203606 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlgp\" (UniqueName: \"kubernetes.io/projected/c6af49c5-244d-47de-bb7f-3d6cfde75f04-kube-api-access-vqlgp\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-trusted-ca\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.203929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.203850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-tmp-dir\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.223571 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.223532 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:48:01.226472 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.226443 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\"" Sep 10 14:48:01.246823 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.246789 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:01.304711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a0946b-0765-4156-b0f8-6759a19ab380-config\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.304711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d8884e-96c9-435b-8150-175a79139143-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tt5n\" (UniqueName: \"kubernetes.io/projected/98ce95dc-ee92-4834-92c4-212c5f260ba9-kube-api-access-6tt5n\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a0946b-0765-4156-b0f8-6759a19ab380-serving-cert\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndt4q\" (UniqueName: \"kubernetes.io/projected/8f9b3eb4-5d07-4088-b242-d23a709660ce-kube-api-access-ndt4q\") pod \"network-check-source-685456655f-chv6f\" (UID: \"8f9b3eb4-5d07-4088-b242-d23a709660ce\") " pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.304957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlgp\" (UniqueName: \"kubernetes.io/projected/c6af49c5-244d-47de-bb7f-3d6cfde75f04-kube-api-access-vqlgp\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-trusted-ca\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.304994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-tmp-dir\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4wk7\" (UniqueName: \"kubernetes.io/projected/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-kube-api-access-q4wk7\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbb4t\" (UniqueName: \"kubernetes.io/projected/e5a0946b-0765-4156-b0f8-6759a19ab380-kube-api-access-tbb4t\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6af49c5-244d-47de-bb7f-3d6cfde75f04-cert\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-config-volume\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrqw\" (UniqueName: \"kubernetes.io/projected/d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32-kube-api-access-pdrqw\") pod \"volume-data-source-validator-695c895bd6-2tvmp\" (UID: \"d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" Sep 10 14:48:01.305267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.305741 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305401 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.305741 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.305432 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:01.305741 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.305495 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:01.805480913 +0000 UTC m=+36.099783020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:01.305741 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4mw\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxpr\" (UniqueName: \"kubernetes.io/projected/b7d8884e-96c9-435b-8150-175a79139143-kube-api-access-5kxpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2sf2\" (UniqueName: \"kubernetes.io/projected/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-kube-api-access-l2sf2\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d8884e-96c9-435b-8150-175a79139143-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305841 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-tmp-dir\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-config\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.305899 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.306075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-serving-cert\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.306075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.305955 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d8884e-96c9-435b-8150-175a79139143-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.306157 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.306139 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:01.306222 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.306209 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:01.806191671 +0000 UTC m=+36.100493778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:01.306419 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.306390 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-trusted-ca\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.306419 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.306421 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:01.306628 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.306434 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:01.306628 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.306476 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.307993 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.306959 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-config-volume\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.307993 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.306486 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:01.806470694 +0000 UTC m=+36.100772802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:01.307993 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.307464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-config\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.309843 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.309730 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a0946b-0765-4156-b0f8-6759a19ab380-config\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.311205 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.311170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d8884e-96c9-435b-8150-175a79139143-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.311313 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.311205 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-serving-cert\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.311313 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.311253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.311313 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.311259 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.311584 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.311562 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6af49c5-244d-47de-bb7f-3d6cfde75f04-cert\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.313642 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.313450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a0946b-0765-4156-b0f8-6759a19ab380-serving-cert\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.315563 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.315539 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.315941 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.315898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4wk7\" (UniqueName: \"kubernetes.io/projected/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-kube-api-access-q4wk7\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.316968 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.316943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlgp\" (UniqueName: \"kubernetes.io/projected/c6af49c5-244d-47de-bb7f-3d6cfde75f04-kube-api-access-vqlgp\") pod \"ingress-canary-hr8bk\" (UID: \"c6af49c5-244d-47de-bb7f-3d6cfde75f04\") " pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.318112 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.318087 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrqw\" (UniqueName: \"kubernetes.io/projected/d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32-kube-api-access-pdrqw\") pod \"volume-data-source-validator-695c895bd6-2tvmp\" (UID: \"d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" Sep 10 14:48:01.318432 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.318408 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbb4t\" (UniqueName: \"kubernetes.io/projected/e5a0946b-0765-4156-b0f8-6759a19ab380-kube-api-access-tbb4t\") pod \"service-ca-operator-859f94c4b-5x6vj\" (UID: \"e5a0946b-0765-4156-b0f8-6759a19ab380\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.318516 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.318483 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tt5n\" (UniqueName: \"kubernetes.io/projected/98ce95dc-ee92-4834-92c4-212c5f260ba9-kube-api-access-6tt5n\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.318987 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.318970 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxpr\" (UniqueName: \"kubernetes.io/projected/b7d8884e-96c9-435b-8150-175a79139143-kube-api-access-5kxpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jhd4p\" (UID: \"b7d8884e-96c9-435b-8150-175a79139143\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.322582 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.322551 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2sf2\" (UniqueName: \"kubernetes.io/projected/7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84-kube-api-access-l2sf2\") pod \"console-operator-85b454874b-xpx2r\" (UID: \"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84\") " pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.322796 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.322775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4mw\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.324169 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.324150 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndt4q\" (UniqueName: \"kubernetes.io/projected/8f9b3eb4-5d07-4088-b242-d23a709660ce-kube-api-access-ndt4q\") pod \"network-check-source-685456655f-chv6f\" (UID: \"8f9b3eb4-5d07-4088-b242-d23a709660ce\") " pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" Sep 10 14:48:01.335920 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.335875 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:01.385631 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.385594 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" Sep 10 14:48:01.411111 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.411074 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" Sep 10 14:48:01.421200 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.421178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:01.428845 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.428737 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" Sep 10 14:48:01.443792 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.443716 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" Sep 10 14:48:01.457392 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.457354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr8bk" Sep 10 14:48:01.691530 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.691281 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp"] Sep 10 14:48:01.695403 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:01.695361 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fd22a3_1b7b_412c_8ae4_bb91cd16cf32.slice/crio-15aee564a7d86e3e41e67c34cad277cb17cac076b3d330c9b540e429ee21fa02 WatchSource:0}: Error finding container 15aee564a7d86e3e41e67c34cad277cb17cac076b3d330c9b540e429ee21fa02: Status 404 returned error can't find the container with id 15aee564a7d86e3e41e67c34cad277cb17cac076b3d330c9b540e429ee21fa02 Sep 10 14:48:01.704016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.703931 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p"] Sep 10 14:48:01.709010 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:01.708977 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d8884e_96c9_435b_8150_175a79139143.slice/crio-42cf85dad879fdf4d847899baa940f00af2aea3e33fb55956d5ad9580fbce971 WatchSource:0}: Error finding container 42cf85dad879fdf4d847899baa940f00af2aea3e33fb55956d5ad9580fbce971: Status 404 returned error can't find the container with id 42cf85dad879fdf4d847899baa940f00af2aea3e33fb55956d5ad9580fbce971 Sep 10 14:48:01.720824 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.720793 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xpx2r"] Sep 10 14:48:01.745475 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.745447 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj"] Sep 10 14:48:01.749047 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:01.748973 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a0946b_0765_4156_b0f8_6759a19ab380.slice/crio-bfdb62c3f1f6301884a421c1b46465906e4c6c239bf1baa5067b74a75c575427 WatchSource:0}: Error finding container bfdb62c3f1f6301884a421c1b46465906e4c6c239bf1baa5067b74a75c575427: Status 404 returned error can't find the container with id bfdb62c3f1f6301884a421c1b46465906e4c6c239bf1baa5067b74a75c575427 Sep 10 14:48:01.787460 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.787430 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-chv6f"] Sep 10 14:48:01.798708 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.798673 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr8bk"] Sep 10 14:48:01.811109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.811078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:01.811109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.811114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:01.811269 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:01.811141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:01.811269 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811244 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:01.811269 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811265 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:01.811397 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811270 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:01.811397 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811240 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:01.811397 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811315 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:02.811300038 +0000 UTC m=+37.105602145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:01.811397 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811328 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:02.811322217 +0000 UTC m=+37.105624324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:01.811397 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:01.811337 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:02.811332245 +0000 UTC m=+37.105634352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:02.223489 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.223450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:48:02.223877 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.223861 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:48:02.226487 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.226455 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\"" Sep 10 14:48:02.226871 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.226490 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:48:02.226871 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.226532 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:48:02.424007 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.423974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" event={"ID":"e5a0946b-0765-4156-b0f8-6759a19ab380","Type":"ContainerStarted","Data":"bfdb62c3f1f6301884a421c1b46465906e4c6c239bf1baa5067b74a75c575427"} Sep 10 14:48:02.425344 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.425312 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" event={"ID":"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84","Type":"ContainerStarted","Data":"ef59060406174f218be942dc39eb1990b069b39731112faf62cc3fca2c77f2ba"} Sep 10 14:48:02.426607 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.426576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" event={"ID":"b7d8884e-96c9-435b-8150-175a79139143","Type":"ContainerStarted","Data":"42cf85dad879fdf4d847899baa940f00af2aea3e33fb55956d5ad9580fbce971"} Sep 10 14:48:02.427917 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.427894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" event={"ID":"d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32","Type":"ContainerStarted","Data":"15aee564a7d86e3e41e67c34cad277cb17cac076b3d330c9b540e429ee21fa02"} Sep 10 14:48:02.819847 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.819804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:02.819847 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.819850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.819886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.819980 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.819984 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.820027 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:04.820014266 +0000 UTC m=+39.114316387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.819989 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.820039 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:04.820033571 +0000 UTC m=+39.114335678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.820044 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:02.820114 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:02.820085 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:04.820067651 +0000 UTC m=+39.114369771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:02.944629 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:02.944587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:03.045672 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.045627 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:03.295360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.295326 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9"] Sep 10 14:48:03.325177 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.325147 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9"] Sep 10 14:48:03.325177 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.325182 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h"] Sep 10 14:48:03.325377 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.325294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.328190 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.328161 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 14:48:03.328341 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.328214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 14:48:03.328341 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.328218 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 14:48:03.328341 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.328218 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jwzdt\"" Sep 10 14:48:03.328472 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.328219 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 14:48:03.341351 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.341323 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx"] Sep 10 14:48:03.341477 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.341364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.343539 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.343510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 14:48:03.358842 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.358812 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h"] Sep 10 14:48:03.359005 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.358862 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx"] Sep 10 14:48:03.359005 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.358943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.361504 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.361481 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 14:48:03.361803 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.361787 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 14:48:03.361870 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.361843 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 14:48:03.361930 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.361843 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 14:48:03.386417 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:03.386384 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9b3eb4_5d07_4088_b242_d23a709660ce.slice/crio-86ffaaffb0a3c6899c57da00753e90c550538b9acb4cb0044a166b0361412d8b WatchSource:0}: Error finding container 86ffaaffb0a3c6899c57da00753e90c550538b9acb4cb0044a166b0361412d8b: Status 404 returned error can't find the container with id 86ffaaffb0a3c6899c57da00753e90c550538b9acb4cb0044a166b0361412d8b Sep 10 14:48:03.387031 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:03.387004 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6af49c5_244d_47de_bb7f_3d6cfde75f04.slice/crio-339d73c1667bc4d25e9bbb2a54c467b7f3a5f5e75f6ce59c514af07ad55033f5 WatchSource:0}: Error finding container 339d73c1667bc4d25e9bbb2a54c467b7f3a5f5e75f6ce59c514af07ad55033f5: Status 404 returned error can't find the container with id 339d73c1667bc4d25e9bbb2a54c467b7f3a5f5e75f6ce59c514af07ad55033f5 Sep 10 14:48:03.425956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.425913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95f77\" (UniqueName: \"kubernetes.io/projected/472b8343-d36f-4146-ae49-50171729f393-kube-api-access-95f77\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.426131 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.426055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472b8343-d36f-4146-ae49-50171729f393-tmp\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.426131 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.426120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhcw\" (UniqueName: \"kubernetes.io/projected/0a7efad2-2638-4830-bbd5-43a74593e698-kube-api-access-jhhcw\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.426222 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.426158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/472b8343-d36f-4146-ae49-50171729f393-klusterlet-config\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.426222 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.426182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a7efad2-2638-4830-bbd5-43a74593e698-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.431836 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.431794 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" event={"ID":"8f9b3eb4-5d07-4088-b242-d23a709660ce","Type":"ContainerStarted","Data":"86ffaaffb0a3c6899c57da00753e90c550538b9acb4cb0044a166b0361412d8b"} Sep 10 14:48:03.432897 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.432872 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr8bk" event={"ID":"c6af49c5-244d-47de-bb7f-3d6cfde75f04","Type":"ContainerStarted","Data":"339d73c1667bc4d25e9bbb2a54c467b7f3a5f5e75f6ce59c514af07ad55033f5"} Sep 10 14:48:03.526864 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.526820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgq9\" (UniqueName: \"kubernetes.io/projected/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-kube-api-access-7vgq9\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.526881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhcw\" (UniqueName: \"kubernetes.io/projected/0a7efad2-2638-4830-bbd5-43a74593e698-kube-api-access-jhhcw\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.527037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.526948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/472b8343-d36f-4146-ae49-50171729f393-klusterlet-config\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.527037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.526980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a7efad2-2638-4830-bbd5-43a74593e698-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.527037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527295 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527295 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-additional-service-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527295 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95f77\" (UniqueName: \"kubernetes.io/projected/472b8343-d36f-4146-ae49-50171729f393-kube-api-access-95f77\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.527295 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527289 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472b8343-d36f-4146-ae49-50171729f393-tmp\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.527459 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.527628 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.527605 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472b8343-d36f-4146-ae49-50171729f393-tmp\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.530411 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.530387 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/472b8343-d36f-4146-ae49-50171729f393-klusterlet-config\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.530520 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.530389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a7efad2-2638-4830-bbd5-43a74593e698-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.534714 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.534689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhcw\" (UniqueName: \"kubernetes.io/projected/0a7efad2-2638-4830-bbd5-43a74593e698-kube-api-access-jhhcw\") pod \"managed-serviceaccount-addon-agent-65f777f796-dr9b9\" (UID: \"0a7efad2-2638-4830-bbd5-43a74593e698\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.535044 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.535016 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95f77\" (UniqueName: \"kubernetes.io/projected/472b8343-d36f-4146-ae49-50171729f393-kube-api-access-95f77\") pod \"klusterlet-addon-workmgr-86db549db8-6vk9h\" (UID: \"472b8343-d36f-4146-ae49-50171729f393\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.628180 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.628180 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-additional-service-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.628360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.628360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgq9\" (UniqueName: \"kubernetes.io/projected/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-kube-api-access-7vgq9\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.628360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.628360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.628303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.629171 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.629133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-additional-service-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.634648 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.631209 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-ca\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.634648 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.631235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.634648 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.631394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.634648 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.631443 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.637472 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.637446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgq9\" (UniqueName: \"kubernetes.io/projected/a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419-kube-api-access-7vgq9\") pod \"cluster-proxy-proxy-agent-668b6df74-fdnsx\" (UID: \"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:03.643780 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.643744 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" Sep 10 14:48:03.651603 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.651576 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:03.667506 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:03.667476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:48:04.010534 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.010437 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx"] Sep 10 14:48:04.022661 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.022592 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h"] Sep 10 14:48:04.024676 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.024620 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9"] Sep 10 14:48:04.029468 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:04.029420 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472b8343_d36f_4146_ae49_50171729f393.slice/crio-57e049804dc006f5a8b51683cb7dc7ad138d3370a7fb82fd37df76dc13d7c362 WatchSource:0}: Error finding container 57e049804dc006f5a8b51683cb7dc7ad138d3370a7fb82fd37df76dc13d7c362: Status 404 returned error can't find the container with id 57e049804dc006f5a8b51683cb7dc7ad138d3370a7fb82fd37df76dc13d7c362 Sep 10 14:48:04.032662 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:04.032226 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d9bfaa_2a71_4058_aa1f_eeea2c3e9419.slice/crio-35d76305b18030af91fdaeb08bc8ba815c5dd25d1a919b64d6fcb5062421cdf8 WatchSource:0}: Error finding container 35d76305b18030af91fdaeb08bc8ba815c5dd25d1a919b64d6fcb5062421cdf8: Status 404 returned error can't find the container with id 35d76305b18030af91fdaeb08bc8ba815c5dd25d1a919b64d6fcb5062421cdf8 Sep 10 14:48:04.034864 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:04.034463 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a7efad2_2638_4830_bbd5_43a74593e698.slice/crio-a45396fdc9c01ee513afaad716cdf89a8ef2550ba6d25fbabd766d1416c2f35c WatchSource:0}: Error finding container a45396fdc9c01ee513afaad716cdf89a8ef2550ba6d25fbabd766d1416c2f35c: Status 404 returned error can't find the container with id a45396fdc9c01ee513afaad716cdf89a8ef2550ba6d25fbabd766d1416c2f35c Sep 10 14:48:04.100110 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.100065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:04.246286 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.246229 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:04.347527 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.347486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:04.439063 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.438734 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" event={"ID":"472b8343-d36f-4146-ae49-50171729f393","Type":"ContainerStarted","Data":"57e049804dc006f5a8b51683cb7dc7ad138d3370a7fb82fd37df76dc13d7c362"} Sep 10 14:48:04.442016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.441980 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerStarted","Data":"35d76305b18030af91fdaeb08bc8ba815c5dd25d1a919b64d6fcb5062421cdf8"} Sep 10 14:48:04.450780 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.450159 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="fe1259c5a1932845bae5d852ed5bde3f5b681cf3232f53e69d20d2fbe993213d" exitCode=0 Sep 10 14:48:04.450780 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.450240 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"fe1259c5a1932845bae5d852ed5bde3f5b681cf3232f53e69d20d2fbe993213d"} Sep 10 14:48:04.455970 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.454926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" event={"ID":"0a7efad2-2638-4830-bbd5-43a74593e698","Type":"ContainerStarted","Data":"a45396fdc9c01ee513afaad716cdf89a8ef2550ba6d25fbabd766d1416c2f35c"} Sep 10 14:48:04.786797 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.786231 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-txkpc"] Sep 10 14:48:04.799031 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.798955 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-txkpc"] Sep 10 14:48:04.799733 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.799567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:04.803556 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.802098 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9z7r6\"" Sep 10 14:48:04.803556 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.802349 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 14:48:04.803556 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.802613 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 14:48:04.838965 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.838925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:04.839152 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.839047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:04.839152 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.839096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:04.839254 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839235 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:04.839254 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839250 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:04.839344 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839314 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:08.839292272 +0000 UTC m=+43.133594398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:04.839772 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839735 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:04.839876 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839808 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:08.839793684 +0000 UTC m=+43.134095805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:04.839936 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839876 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:04.839936 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:04.839911 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:08.839899825 +0000 UTC m=+43.134201953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:04.940297 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.940077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:04.940297 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.940167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e2a41a6f-a85e-4b46-b718-4a0ec8974874-nginx-conf\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:04.948054 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:04.947989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:05.041094 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.041008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:05.041094 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.041086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e2a41a6f-a85e-4b46-b718-4a0ec8974874-nginx-conf\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:05.042282 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.041837 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e2a41a6f-a85e-4b46-b718-4a0ec8974874-nginx-conf\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:05.042282 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:05.041964 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:05.042282 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:05.042021 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:05.542002783 +0000 UTC m=+39.836304895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:05.364770 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.364656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:05.423397 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.423316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:05.468148 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.467215 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerStarted","Data":"d0003e150b7be4872a57602efd5bfbfa67360a3b7e38e74a9abd52d3925b2022"} Sep 10 14:48:05.546314 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:05.546276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:05.546469 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:05.546453 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:05.546966 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:05.546529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:06.546508886 +0000 UTC m=+40.840811014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:06.136373 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.136323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:06.187151 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.187098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:06.479835 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.478961 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" event={"ID":"d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32","Type":"ContainerStarted","Data":"80d564f1137b325f41229c9fc3db25180a5d33bec1af0b456165124904d02b50"} Sep 10 14:48:06.489831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.489654 2581 generic.go:356] "Generic (PLEG): container finished" podID="7009b52c-fa14-4627-af75-b0af34939d26" containerID="d0003e150b7be4872a57602efd5bfbfa67360a3b7e38e74a9abd52d3925b2022" exitCode=0 Sep 10 14:48:06.489831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.489700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerDied","Data":"d0003e150b7be4872a57602efd5bfbfa67360a3b7e38e74a9abd52d3925b2022"} Sep 10 14:48:06.496673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.496309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-2tvmp" podStartSLOduration=34.696262744 podStartE2EDuration="38.49629222s" totalImagesPullingTime="3.800029476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:06.495069285 +0000 UTC m=+40.789371417" watchObservedRunningTime="2026-09-10 14:48:06.49629222 +0000 UTC m=+40.790594349" Sep 10 14:48:06.553094 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.553048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:06.557269 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.557223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:06.557442 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:06.557427 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:06.557514 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:06.557493 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:08.557472931 +0000 UTC m=+42.851775060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:06.661319 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.661268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:06.756336 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:06.756179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:07.289653 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:07.289587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:07.389906 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:07.389827 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:07.488317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:07.488232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:07.742211 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:07.742102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:07.802009 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:07.801914 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:08.555668 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.555616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:08.581778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.581722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:08.581960 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.581890 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:08.581960 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.581956 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:12.581937695 +0000 UTC m=+46.876239804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:08.885400 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.885342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:08.885588 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.885424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:08.885588 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.885460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:08.885588 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885537 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:08.885722 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885603 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:08.885722 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885697 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:08.885722 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885707 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:08.885880 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885608 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:16.88559128 +0000 UTC m=+51.179893386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:08.885880 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885775 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:16.885735924 +0000 UTC m=+51.180038037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:08.885880 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:08.885792 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:16.885782232 +0000 UTC m=+51.180084345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:08.945216 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:08.945176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:09.042390 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:09.042354 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:11.177171 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:11.177138 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:11.751992 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:11.751926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:12.616967 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:12.616924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:12.617434 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:12.617030 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:12.617434 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:12.617113 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:20.617092186 +0000 UTC m=+54.911394296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:15.445387 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:15.445355 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:15.555359 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:15.555293 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:15.724733 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:15.724504 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:15.779217 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:15.779181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:16.953493 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:16.953453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:16.953509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:16.953529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953619 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953627 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953671 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953690 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d57f78776-7wqpj: secret "image-registry-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953679 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls podName:57af0fe3-cfc0-4fad-a725-fae7deb33ab0 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:32.953662446 +0000 UTC m=+67.247964553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls") pod "dns-default-lmd2s" (UID: "57af0fe3-cfc0-4fad-a725-fae7deb33ab0") : secret "dns-default-metrics-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953744 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls podName:98ce95dc-ee92-4834-92c4-212c5f260ba9 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:32.95372978 +0000 UTC m=+67.248031889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-tgqt8" (UID: "98ce95dc-ee92-4834-92c4-212c5f260ba9") : secret "samples-operator-tls" not found Sep 10 14:48:16.953923 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:16.953847 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls podName:8c4282dd-0e3a-407e-a5fa-923831d9684e nodeName:}" failed. No retries permitted until 2026-09-10 14:48:32.953835438 +0000 UTC m=+67.248137552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls") pod "image-registry-d57f78776-7wqpj" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e") : secret "image-registry-tls" not found Sep 10 14:48:17.525429 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.525388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" event={"ID":"472b8343-d36f-4146-ae49-50171729f393","Type":"ContainerStarted","Data":"6d2adbd19483e9faa7fbb3eecd95909a486db40e32cd1bbcc761fad2c236db7d"} Sep 10 14:48:17.526012 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.525986 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:17.527625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.527593 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" event={"ID":"e5a0946b-0765-4156-b0f8-6759a19ab380","Type":"ContainerStarted","Data":"881323adb07c11e5f3abfad098878a58c721567f0ccff788cf682484c09ad461"} Sep 10 14:48:17.528367 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.528344 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" Sep 10 14:48:17.530142 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.530116 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/0.log" Sep 10 14:48:17.530250 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.530156 2581 generic.go:356] "Generic (PLEG): container finished" podID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" containerID="80af0ec9b74c6b32dd8e7690a07c312ee2b20017fdff1e65bed4a5bcb7539f44" exitCode=255 Sep 10 14:48:17.530464 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.530389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" event={"ID":"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84","Type":"ContainerDied","Data":"80af0ec9b74c6b32dd8e7690a07c312ee2b20017fdff1e65bed4a5bcb7539f44"} Sep 10 14:48:17.530542 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.530522 2581 scope.go:117] "RemoveContainer" containerID="80af0ec9b74c6b32dd8e7690a07c312ee2b20017fdff1e65bed4a5bcb7539f44" Sep 10 14:48:17.532593 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.532143 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" event={"ID":"b7d8884e-96c9-435b-8150-175a79139143","Type":"ContainerStarted","Data":"3ff5e794b1e8f244790051617ed6819c0f81c27ecbf45969fb5d7b116fd425bb"} Sep 10 14:48:17.534507 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.534482 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" event={"ID":"8f9b3eb4-5d07-4088-b242-d23a709660ce","Type":"ContainerStarted","Data":"b9e4e2322c8262c01e1bf020ccaa77d9c0dc0ceb012b20b6e1e8878c8340460a"} Sep 10 14:48:17.543420 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.543364 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerStarted","Data":"2f10f7e92ea6d4576b1654fb6c9cfd4d1966cb5a5fdc2483029edd0731400af9"} Sep 10 14:48:17.543621 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.543450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerStarted","Data":"168d917ef145a0116a28546bdcc3c572b3c6b9e3d420a6d90561903526200333"} Sep 10 14:48:17.543621 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.543469 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerStarted","Data":"4ba5027bf1a9c4a3e89f2023b73a3042a535787496c89c2737c8052c92bb037f"} Sep 10 14:48:17.548458 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.548430 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" event={"ID":"7009b52c-fa14-4627-af75-b0af34939d26","Type":"ContainerStarted","Data":"15c1d8dc232007b329e75339ddc1cec375d6c0c8efaeb1c8a02a34c593056a8b"} Sep 10 14:48:17.552548 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.552497 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-86db549db8-6vk9h" podStartSLOduration=1.648550626 podStartE2EDuration="14.552480521s" totalImagesPullingTime="12.903929895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.541798326 +0000 UTC m=+51.836100455" watchObservedRunningTime="2026-09-10 14:48:17.552480521 +0000 UTC m=+51.846782644" Sep 10 14:48:17.554368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.554341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr8bk" event={"ID":"c6af49c5-244d-47de-bb7f-3d6cfde75f04","Type":"ContainerStarted","Data":"35c2b232c60c59506f09a892ff692e6963bcc92ab7adf983ea0738fb06ac56bb"} Sep 10 14:48:17.556477 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.556445 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" event={"ID":"0a7efad2-2638-4830-bbd5-43a74593e698","Type":"ContainerStarted","Data":"36ed677ed116b99b6da43ec4e04bc5d0b43ae6930412e5b5390aad6709d7cda5"} Sep 10 14:48:17.576671 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.576603 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" podStartSLOduration=34.474790676 podStartE2EDuration="49.576580855s" totalImagesPullingTime="15.101790179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.576184731 +0000 UTC m=+51.870486861" watchObservedRunningTime="2026-09-10 14:48:17.576580855 +0000 UTC m=+51.870882983" Sep 10 14:48:17.614880 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.614613 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" podStartSLOduration=34.428651967 podStartE2EDuration="49.614595092s" totalImagesPullingTime="15.185943125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.613347119 +0000 UTC m=+51.907649248" watchObservedRunningTime="2026-09-10 14:48:17.614595092 +0000 UTC m=+51.908897220" Sep 10 14:48:17.629915 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.629869 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-chv6f" podStartSLOduration=26.120652117 podStartE2EDuration="39.629852072s" totalImagesPullingTime="13.509199955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.629169602 +0000 UTC m=+51.923471731" watchObservedRunningTime="2026-09-10 14:48:17.629852072 +0000 UTC m=+51.924154200" Sep 10 14:48:17.653469 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.653414 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hvfqw" podStartSLOduration=17.279752227 podStartE2EDuration="50.653393609s" totalImagesPullingTime="33.373641382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.65220886 +0000 UTC m=+51.946510991" watchObservedRunningTime="2026-09-10 14:48:17.653393609 +0000 UTC m=+51.947695739" Sep 10 14:48:17.690264 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.690193 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hr8bk" podStartSLOduration=3.181385039 podStartE2EDuration="16.690170599s" totalImagesPullingTime="13.50878556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.669617597 +0000 UTC m=+51.963919726" watchObservedRunningTime="2026-09-10 14:48:17.690170599 +0000 UTC m=+51.984472729" Sep 10 14:48:17.705612 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.705554 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65f777f796-dr9b9" podStartSLOduration=1.823297019 podStartE2EDuration="14.705530766s" totalImagesPullingTime="12.882233747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.704693133 +0000 UTC m=+51.998995263" watchObservedRunningTime="2026-09-10 14:48:17.705530766 +0000 UTC m=+51.999832895" Sep 10 14:48:17.706247 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.706197 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" podStartSLOduration=1.815730319 podStartE2EDuration="14.706183089s" totalImagesPullingTime="12.89045277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:17.689850582 +0000 UTC m=+51.984152711" watchObservedRunningTime="2026-09-10 14:48:17.706183089 +0000 UTC m=+52.000485221" Sep 10 14:48:17.754652 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.754596 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:17.814570 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:17.814531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.186815 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.186750 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.427509 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.427473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.562929 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.562902 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:48:18.563373 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.563353 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/0.log" Sep 10 14:48:18.563484 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.563397 2581 generic.go:356] "Generic (PLEG): container finished" podID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" containerID="bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d" exitCode=255 Sep 10 14:48:18.563533 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.563499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" event={"ID":"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84","Type":"ContainerDied","Data":"bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d"} Sep 10 14:48:18.563590 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.563558 2581 scope.go:117] "RemoveContainer" containerID="80af0ec9b74c6b32dd8e7690a07c312ee2b20017fdff1e65bed4a5bcb7539f44" Sep 10 14:48:18.563774 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.563735 2581 scope.go:117] "RemoveContainer" containerID="bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d" Sep 10 14:48:18.563989 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:18.563960 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xpx2r_openshift-console-operator(7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84)\"" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" podUID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" Sep 10 14:48:18.611271 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.611229 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.675853 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.675824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.827966 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.827872 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:18.921020 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:18.920982 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:19.018161 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:19.018120 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:19.568015 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:19.567988 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:48:19.568446 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:19.568330 2581 scope.go:117] "RemoveContainer" containerID="bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d" Sep 10 14:48:19.568506 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:19.568475 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xpx2r_openshift-console-operator(7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84)\"" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" podUID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" Sep 10 14:48:20.512048 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:20.512013 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:20.568142 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:20.568109 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:20.691364 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:20.691300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:20.691550 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:20.691471 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 14:48:20.691646 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:20.691552 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert podName:e2a41a6f-a85e-4b46-b718-4a0ec8974874 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:36.691531272 +0000 UTC m=+70.985833403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-txkpc" (UID: "e2a41a6f-a85e-4b46-b718-4a0ec8974874") : secret "networking-console-plugin-cert" not found Sep 10 14:48:20.888382 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:20.888294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:20.949616 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:20.949583 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:21.410878 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:21.410831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:21.421651 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:21.421610 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:21.421811 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:21.421687 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:21.422055 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:21.422042 2581 scope.go:117] "RemoveContainer" containerID="bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d" Sep 10 14:48:21.422208 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:21.422189 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xpx2r_openshift-console-operator(7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84)\"" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" podUID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" Sep 10 14:48:24.391913 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:24.391881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:24.848660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:24.848623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:24.986833 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:24.986800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.090866 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.090831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.338114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.338076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.438810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.438774 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.569336 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.569299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.643562 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.643475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.793887 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:25.793851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:26.871102 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:26.871048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:26.981506 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:26.981470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:27.382660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:27.382626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:27.588251 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:27.588211 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:27.903608 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:27.903563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:28.190523 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:28.190431 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:28.422407 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:28.422380 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zg967" Sep 10 14:48:28.586938 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:28.586891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:28.781175 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:28.781130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:28.982625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:28.982531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:29.289813 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:29.289772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:29.390000 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:29.389965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:29.984304 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:29.984273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:30.039460 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.039433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:30.183259 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.183225 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:30.582497 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.582464 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:30.786003 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.785967 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:30.982442 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.982341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:48:30.982442 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.982413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:48:30.985039 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.985019 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:48:30.985357 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.985030 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:48:30.993265 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:30.993233 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 14:48:30.993360 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:30.993314 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs podName:238df29e-1e13-4956-9f42-ed8340fd572f nodeName:}" failed. No retries permitted until 2026-09-10 14:49:34.993295403 +0000 UTC m=+129.287597511 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs") pod "network-metrics-daemon-bkhbf" (UID: "238df29e-1e13-4956-9f42-ed8340fd572f") : secret "metrics-daemon-secret" not found Sep 10 14:48:30.996690 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:30.996663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d109b72e-e301-4f56-9c25-296228e6257c-original-pull-secret\") pod \"global-pull-secret-syncer-nhsnn\" (UID: \"d109b72e-e301-4f56-9c25-296228e6257c\") " pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:48:31.037041 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.037002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhsnn" Sep 10 14:48:31.183039 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.183011 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:31.183956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.183933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:48:31.186881 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.186857 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2j9x\" (UniqueName: \"kubernetes.io/projected/a25b18d9-742c-4589-bdd0-c90899b64ad9-kube-api-access-k2j9x\") pod \"network-check-target-jxmnh\" (UID: \"a25b18d9-742c-4589-bdd0-c90899b64ad9\") " pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:48:31.237233 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.237205 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\"" Sep 10 14:48:31.244681 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.244653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:48:31.307840 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.307809 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhsnn"] Sep 10 14:48:31.310021 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:31.309991 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd109b72e_e301_4f56_9c25_296228e6257c.slice/crio-56ecc840f232ea20ad1987723d7131943ed64ecd68f8f43b7649e9e81bab4349 WatchSource:0}: Error finding container 56ecc840f232ea20ad1987723d7131943ed64ecd68f8f43b7649e9e81bab4349: Status 404 returned error can't find the container with id 56ecc840f232ea20ad1987723d7131943ed64ecd68f8f43b7649e9e81bab4349 Sep 10 14:48:31.473538 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.473508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:31.510976 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.510890 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxmnh"] Sep 10 14:48:31.515142 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:31.515114 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25b18d9_742c_4589_bdd0_c90899b64ad9.slice/crio-115740d8c78c4daaa3abf1770b50bb62d9522e0e354fe3f05a6f38990197cfc9 WatchSource:0}: Error finding container 115740d8c78c4daaa3abf1770b50bb62d9522e0e354fe3f05a6f38990197cfc9: Status 404 returned error can't find the container with id 115740d8c78c4daaa3abf1770b50bb62d9522e0e354fe3f05a6f38990197cfc9 Sep 10 14:48:31.609622 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.609584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxmnh" event={"ID":"a25b18d9-742c-4589-bdd0-c90899b64ad9","Type":"ContainerStarted","Data":"c39ea093cb56b45d5118244397ba15f70c42d3b7dcda81ff22f78c0e2e413d55"} Sep 10 14:48:31.609828 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.609637 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxmnh" event={"ID":"a25b18d9-742c-4589-bdd0-c90899b64ad9","Type":"ContainerStarted","Data":"115740d8c78c4daaa3abf1770b50bb62d9522e0e354fe3f05a6f38990197cfc9"} Sep 10 14:48:31.609828 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.609662 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:48:31.610863 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.610830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhsnn" event={"ID":"d109b72e-e301-4f56-9c25-296228e6257c","Type":"ContainerStarted","Data":"56ecc840f232ea20ad1987723d7131943ed64ecd68f8f43b7649e9e81bab4349"} Sep 10 14:48:31.624935 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.624882 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jxmnh" podStartSLOduration=64.624864309 podStartE2EDuration="1m4.624864309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:31.623001541 +0000 UTC m=+65.917303670" watchObservedRunningTime="2026-09-10 14:48:31.624864309 +0000 UTC m=+65.919166439" Sep 10 14:48:31.790368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:31.790247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:32.215087 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:32.215000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:32.589797 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:32.589739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:32.835147 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:32.835098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:33.002655 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.002613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:33.002855 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.002688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:33.002855 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.002722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:33.005773 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.005722 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57af0fe3-cfc0-4fad-a725-fae7deb33ab0-metrics-tls\") pod \"dns-default-lmd2s\" (UID: \"57af0fe3-cfc0-4fad-a725-fae7deb33ab0\") " pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:33.005927 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.005808 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98ce95dc-ee92-4834-92c4-212c5f260ba9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-tgqt8\" (UID: \"98ce95dc-ee92-4834-92c4-212c5f260ba9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:33.005927 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.005825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"image-registry-d57f78776-7wqpj\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:33.203016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.202978 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v7wcd\"" Sep 10 14:48:33.210928 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.210894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" Sep 10 14:48:33.238313 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.238285 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgzsd\"" Sep 10 14:48:33.246334 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.246300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:33.252898 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.252826 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7c4td\"" Sep 10 14:48:33.261794 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.261682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:33.268722 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.268689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:33.497664 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.497633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8"] Sep 10 14:48:33.534890 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.534868 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:48:33.537366 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:33.537339 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4282dd_0e3a_407e_a5fa_923831d9684e.slice/crio-39c3b5010b7dfee38b242716fd57397669e85db36613307aa35145db82b62794 WatchSource:0}: Error finding container 39c3b5010b7dfee38b242716fd57397669e85db36613307aa35145db82b62794: Status 404 returned error can't find the container with id 39c3b5010b7dfee38b242716fd57397669e85db36613307aa35145db82b62794 Sep 10 14:48:33.561418 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.561385 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmd2s"] Sep 10 14:48:33.564180 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:33.564147 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af0fe3_cfc0_4fad_a725_fae7deb33ab0.slice/crio-ab2819a09acfbe7edd40287f2f31179e1e0cccd70c1b9684d70d0835069d9ff8 WatchSource:0}: Error finding container ab2819a09acfbe7edd40287f2f31179e1e0cccd70c1b9684d70d0835069d9ff8: Status 404 returned error can't find the container with id ab2819a09acfbe7edd40287f2f31179e1e0cccd70c1b9684d70d0835069d9ff8 Sep 10 14:48:33.619598 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.619565 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmd2s" event={"ID":"57af0fe3-cfc0-4fad-a725-fae7deb33ab0","Type":"ContainerStarted","Data":"ab2819a09acfbe7edd40287f2f31179e1e0cccd70c1b9684d70d0835069d9ff8"} Sep 10 14:48:33.621125 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.621092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" event={"ID":"8c4282dd-0e3a-407e-a5fa-923831d9684e","Type":"ContainerStarted","Data":"39c3b5010b7dfee38b242716fd57397669e85db36613307aa35145db82b62794"} Sep 10 14:48:33.622543 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.622515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" event={"ID":"98ce95dc-ee92-4834-92c4-212c5f260ba9","Type":"ContainerStarted","Data":"d42e43dfa046da82a61706f5f96fad680f8fafda38cc5440c77b7edf4cbd8f63"} Sep 10 14:48:33.784931 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.784834 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:33.988742 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:33.988701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:34.224291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.224206 2581 scope.go:117] "RemoveContainer" containerID="bc1c44111271d59f8bba93e5de36f328dda4e374900cf2ab7d7bc46e91c9fc8d" Sep 10 14:48:34.585124 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.585082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:34.630042 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.630010 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:48:34.630214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.630151 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" event={"ID":"7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84","Type":"ContainerStarted","Data":"08db59a80386f472964c8c4e562281e9238815585347c06cab92c70ccfc7213e"} Sep 10 14:48:34.630883 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.630678 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:34.633575 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.633073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" event={"ID":"8c4282dd-0e3a-407e-a5fa-923831d9684e","Type":"ContainerStarted","Data":"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b"} Sep 10 14:48:34.633575 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.633220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:34.651446 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.651267 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" podStartSLOduration=51.480174017 podStartE2EDuration="1m6.651244144s" totalImagesPullingTime="15.171070127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:34.650576653 +0000 UTC m=+68.944878787" watchObservedRunningTime="2026-09-10 14:48:34.651244144 +0000 UTC m=+68.945546275" Sep 10 14:48:34.669170 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.669101 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" podStartSLOduration=47.669078054 podStartE2EDuration="47.669078054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:34.668147681 +0000 UTC m=+68.962449811" watchObservedRunningTime="2026-09-10 14:48:34.669078054 +0000 UTC m=+68.963380182" Sep 10 14:48:34.990460 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:34.990415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:35.185352 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:35.185316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:35.385195 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:35.385105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:35.631067 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:35.631019 2581 patch_prober.go:28] interesting pod/console-operator-85b454874b-xpx2r container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 14:48:35.631536 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:35.631106 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" podUID="7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 10 14:48:35.830534 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:35.830501 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-xpx2r" Sep 10 14:48:36.010087 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.010043 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.111117 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.111011 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.210282 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.210248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.311278 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.311241 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.411914 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.411823 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.512086 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.512054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.692544 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.692438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.737511 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.737463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:36.740488 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.740456 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a41a6f-a85e-4b46-b718-4a0ec8974874-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-txkpc\" (UID: \"e2a41a6f-a85e-4b46-b718-4a0ec8974874\") " pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:36.913890 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.913835 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.918798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.918772 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9z7r6\"" Sep 10 14:48:36.927738 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:36.927711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" Sep 10 14:48:37.015303 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.015270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:37.289651 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.289580 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-txkpc"] Sep 10 14:48:37.292364 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:37.292332 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a41a6f_a85e_4b46_b718_4a0ec8974874.slice/crio-b5080f4763d0aec26d2c1d3523d24ecaf3f688363e3db5cd9426250ed439c2fc WatchSource:0}: Error finding container b5080f4763d0aec26d2c1d3523d24ecaf3f688363e3db5cd9426250ed439c2fc: Status 404 returned error can't find the container with id b5080f4763d0aec26d2c1d3523d24ecaf3f688363e3db5cd9426250ed439c2fc Sep 10 14:48:37.646672 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.646577 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhsnn" event={"ID":"d109b72e-e301-4f56-9c25-296228e6257c","Type":"ContainerStarted","Data":"8f252a9e542844aec15d1a99702f3434b859d7aa880014cfcbb852f36c2a95ae"} Sep 10 14:48:37.648495 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.648470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" event={"ID":"98ce95dc-ee92-4834-92c4-212c5f260ba9","Type":"ContainerStarted","Data":"5535f0761baeed62750f544d32eea7ee378d90b6e599b5bae2a4953cd24d576a"} Sep 10 14:48:37.648644 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.648527 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" event={"ID":"98ce95dc-ee92-4834-92c4-212c5f260ba9","Type":"ContainerStarted","Data":"33128ce3742ba870b81c0b5de841e63305ef58e5fa833538897e8f8e05d2d3f3"} Sep 10 14:48:37.649724 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.649688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" event={"ID":"e2a41a6f-a85e-4b46-b718-4a0ec8974874","Type":"ContainerStarted","Data":"b5080f4763d0aec26d2c1d3523d24ecaf3f688363e3db5cd9426250ed439c2fc"} Sep 10 14:48:37.651567 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.651545 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmd2s" event={"ID":"57af0fe3-cfc0-4fad-a725-fae7deb33ab0","Type":"ContainerStarted","Data":"677918d58f8e9c23b3833495a50b31adc508aee541ebf414b24f1b4b81625153"} Sep 10 14:48:37.651700 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.651586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmd2s" event={"ID":"57af0fe3-cfc0-4fad-a725-fae7deb33ab0","Type":"ContainerStarted","Data":"9a7d8fda04ca46c043ccadd1f2620e389f68fd1f335302a948ec0b774351cf45"} Sep 10 14:48:37.651700 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.651620 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:37.663443 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.663389 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nhsnn" podStartSLOduration=64.950870517 podStartE2EDuration="1m10.663372269s" totalImagesPullingTime="5.712501752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:37.662383181 +0000 UTC m=+71.956685312" watchObservedRunningTime="2026-09-10 14:48:37.663372269 +0000 UTC m=+71.957674391" Sep 10 14:48:37.676956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.676904 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-tgqt8" podStartSLOduration=66.226232714 podStartE2EDuration="1m9.676887337s" totalImagesPullingTime="3.450654623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:37.675581036 +0000 UTC m=+71.969883177" watchObservedRunningTime="2026-09-10 14:48:37.676887337 +0000 UTC m=+71.971189504" Sep 10 14:48:37.690561 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.690483 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lmd2s" podStartSLOduration=33.184896736 podStartE2EDuration="36.690459001s" totalImagesPullingTime="3.505562265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:37.690294647 +0000 UTC m=+71.984596777" watchObservedRunningTime="2026-09-10 14:48:37.690459001 +0000 UTC m=+71.984761132" Sep 10 14:48:37.902191 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:37.902103 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:38.098633 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.098545 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:38.319217 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.319179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:38.656970 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.656931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" event={"ID":"e2a41a6f-a85e-4b46-b718-4a0ec8974874","Type":"ContainerStarted","Data":"82c53289d89856e80d164bfeeb3935c7be7f48c1ed8964b081f93f07807b833b"} Sep 10 14:48:38.673949 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.673896 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-txkpc" podStartSLOduration=33.408403207 podStartE2EDuration="34.67387964s" totalImagesPullingTime="1.265476433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:38.672780267 +0000 UTC m=+72.967082387" watchObservedRunningTime="2026-09-10 14:48:38.67387964 +0000 UTC m=+72.968181806" Sep 10 14:48:38.721455 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.721423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:38.808660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:38.808630 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:40.301169 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:40.301133 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:48:40.410133 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:40.410097 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:40.570668 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:40.570591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:40.678381 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:40.678345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.060006 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:43.059969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.157661 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:43.157629 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.104097 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.104065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.231315 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.231255 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.332840 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.332802 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vlqpb"] Sep 10 14:48:44.364300 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.364195 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.393110 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.393069 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vlqpb"] Sep 10 14:48:44.393251 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.393197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.395767 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.395728 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 14:48:44.395908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.395796 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 14:48:44.395908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.395818 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g76rw\"" Sep 10 14:48:44.395908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.395843 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 14:48:44.396622 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.396607 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 14:48:44.431914 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.431873 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.487823 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.487749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:44.500838 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.500793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mcx\" (UniqueName: \"kubernetes.io/projected/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-api-access-l8mcx\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.501010 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.500853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-crio-socket\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.501010 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.500934 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-data-volume\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.501010 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.500957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.501010 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.500975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601672 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-data-volume\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601672 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mcx\" (UniqueName: \"kubernetes.io/projected/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-api-access-l8mcx\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-crio-socket\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:44.601815 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:44.601826 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.601859 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-crio-socket\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:44.601888 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.101871579 +0000 UTC m=+79.396173686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:44.601965 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:44.601903 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.101896277 +0000 UTC m=+79.396198384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : configmap "kube-rbac-proxy" not found Sep 10 14:48:44.602230 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.602029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-data-volume\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:44.610628 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:44.610602 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mcx\" (UniqueName: \"kubernetes.io/projected/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-api-access-l8mcx\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:45.106098 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.106061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:45.106098 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.106104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:45.106502 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:45.106222 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:45.106502 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:45.106235 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:45.106502 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:45.106302 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.10628297 +0000 UTC m=+80.400585091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:45.106502 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:45.106317 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.1063104 +0000 UTC m=+80.400612507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : configmap "kube-rbac-proxy" not found Sep 10 14:48:45.269148 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.269116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:45.313408 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.313370 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmd2s_57af0fe3-cfc0-4fad-a725-fae7deb33ab0/dns/0.log" Sep 10 14:48:45.329174 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.329131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:45.449453 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.449363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:45.512454 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:45.512421 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmd2s_57af0fe3-cfc0-4fad-a725-fae7deb33ab0/kube-rbac-proxy/0.log" Sep 10 14:48:46.117978 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:46.117930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:46.117978 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:46.117981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:46.118516 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:46.118093 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:46.118516 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:46.118123 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:46.118516 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:46.118173 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.118156768 +0000 UTC m=+82.412458875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:46.118516 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:46.118188 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.118182004 +0000 UTC m=+82.412484111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : configmap "kube-rbac-proxy" not found Sep 10 14:48:46.180738 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:46.180698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:46.313923 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:46.313894 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9gr4p_33eedfe2-81ba-4f3a-ba9c-5206a372afd8/dns-node-resolver/0.log" Sep 10 14:48:47.113752 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.113636 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d57f78776-7wqpj_8c4282dd-0e3a-407e-a5fa-923831d9684e/registry/0.log" Sep 10 14:48:47.270120 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.270080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:47.333425 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.333389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:47.513377 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.513349 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dp6lf_5d602dc7-5a71-455b-bc86-7f096b2cb9c8/node-ca/0.log" Sep 10 14:48:47.598083 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.598041 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:47.659908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:47.659880 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmd2s" Sep 10 14:48:48.132939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:48.132892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:48.132939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:48.132935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:48.133155 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:48.133042 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:48.133155 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:48.133050 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:48.133155 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:48.133098 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.133084312 +0000 UTC m=+86.427386419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : configmap "kube-rbac-proxy" not found Sep 10 14:48:48.133155 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:48.133112 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls podName:e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.133105751 +0000 UTC m=+86.427407858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlqpb" (UID: "e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:48.313424 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:48.313398 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hr8bk_c6af49c5-244d-47de-bb7f-3d6cfde75f04/serve-healthcheck-canary/0.log" Sep 10 14:48:48.640721 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:48.640677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:49.024264 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:49.024231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:50.307306 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:50.307279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:48:50.561430 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:50.561326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:51.172781 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:51.172727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:51.229470 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:51.229436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:51.416007 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:51.415974 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:51.477036 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:51.476948 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:52.115462 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.115417 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:52.168593 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.168550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:52.168593 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.168591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:52.169109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.169090 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:52.171131 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.171111 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlqpb\" (UID: \"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b\") " pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:52.203181 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.203129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vlqpb" Sep 10 14:48:52.217313 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.217280 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:52.475956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.475880 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vlqpb"] Sep 10 14:48:52.479160 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:52.479129 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e0019b_c19f_4ee9_ae9b_c9743a75fd0b.slice/crio-a27f76ec163c23eb629649692fdd56a22e1ebc183344a110a805ff0fe9745908 WatchSource:0}: Error finding container a27f76ec163c23eb629649692fdd56a22e1ebc183344a110a805ff0fe9745908: Status 404 returned error can't find the container with id a27f76ec163c23eb629649692fdd56a22e1ebc183344a110a805ff0fe9745908 Sep 10 14:48:52.709732 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.709698 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlqpb" event={"ID":"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b","Type":"ContainerStarted","Data":"f3ae1bc0d4a642cb63c33dfd98832fa505b1ae0824985bda45c01148f04da579"} Sep 10 14:48:52.709927 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:52.709780 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlqpb" event={"ID":"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b","Type":"ContainerStarted","Data":"a27f76ec163c23eb629649692fdd56a22e1ebc183344a110a805ff0fe9745908"} Sep 10 14:48:53.139359 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.139328 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.239297 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.239254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.426477 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.426382 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.530551 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.530502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.635747 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.635707 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.673028 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.672989 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zc59w"] Sep 10 14:48:53.675497 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.675470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.677874 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.677813 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 14:48:53.678220 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.678186 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 14:48:53.678322 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.678237 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 14:48:53.678884 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.678868 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 14:48:53.678994 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.678883 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 14:48:53.678994 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.678919 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c5gfh\"" Sep 10 14:48:53.679105 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.679023 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 14:48:53.720977 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.720945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.780868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.780828 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-sys\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.780868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.780867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-root\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.780889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-textfile\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.780940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-metrics-client-ca\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.780982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.781032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781075 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.781061 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781226 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.781116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-wtmp\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.781226 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.781139 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgv5z\" (UniqueName: \"kubernetes.io/projected/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-kube-api-access-vgv5z\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.818512 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.818474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:53.881876 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-sys\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.881876 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-root\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881907 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-textfile\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-metrics-client-ca\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881956 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-sys\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.881975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-root\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-wtmp\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882495 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgv5z\" (UniqueName: \"kubernetes.io/projected/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-kube-api-access-vgv5z\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882495 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:53.882151 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 14:48:53.882495 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:48:53.882234 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls podName:e21bca5a-adea-4fc4-865d-aa2d5b68d18a nodeName:}" failed. No retries permitted until 2026-09-10 14:48:54.382192138 +0000 UTC m=+88.676494254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls") pod "node-exporter-zc59w" (UID: "e21bca5a-adea-4fc4-865d-aa2d5b68d18a") : secret "node-exporter-tls" not found Sep 10 14:48:53.882495 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882297 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-textfile\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882495 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882440 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-wtmp\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882772 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882701 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-metrics-client-ca\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.882861 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.882841 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.884447 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.884425 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.890595 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.890561 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgv5z\" (UniqueName: \"kubernetes.io/projected/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-kube-api-access-vgv5z\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:53.919111 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:53.919077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:54.129466 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.129428 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:54.252495 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.252461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:54.386016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.385928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:54.388906 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.388870 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21bca5a-adea-4fc4-865d-aa2d5b68d18a-node-exporter-tls\") pod \"node-exporter-zc59w\" (UID: \"e21bca5a-adea-4fc4-865d-aa2d5b68d18a\") " pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:54.585654 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.585616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zc59w" Sep 10 14:48:54.596910 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:48:54.596875 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bca5a_adea_4fc4_865d_aa2d5b68d18a.slice/crio-bedf88e2cf83bb364f4bb13211b05a0223d4b90d6e76ca7af33ad996db044242 WatchSource:0}: Error finding container bedf88e2cf83bb364f4bb13211b05a0223d4b90d6e76ca7af33ad996db044242: Status 404 returned error can't find the container with id bedf88e2cf83bb364f4bb13211b05a0223d4b90d6e76ca7af33ad996db044242 Sep 10 14:48:54.633871 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.633818 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:54.712676 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.712588 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:54.719269 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.719237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc59w" event={"ID":"e21bca5a-adea-4fc4-865d-aa2d5b68d18a","Type":"ContainerStarted","Data":"bedf88e2cf83bb364f4bb13211b05a0223d4b90d6e76ca7af33ad996db044242"} Sep 10 14:48:54.721441 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.721410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlqpb" event={"ID":"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b","Type":"ContainerStarted","Data":"e09ef2ad93c16c7647a7cbf138bb943feaa560bbf55b46375073955755be7afb"} Sep 10 14:48:54.834418 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:54.834379 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:55.728182 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:55.728078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlqpb" event={"ID":"e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b","Type":"ContainerStarted","Data":"d2897f26376dd995a8c99e5d2b0cb2518481a2c2ce7cb7375b53ffbcbfe7cafa"} Sep 10 14:48:55.745853 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:55.745798 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vlqpb" podStartSLOduration=9.088950145 podStartE2EDuration="11.745781755s" totalImagesPullingTime="2.65683161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:55.743885253 +0000 UTC m=+90.038187372" watchObservedRunningTime="2026-09-10 14:48:55.745781755 +0000 UTC m=+90.040083883" Sep 10 14:48:56.576686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:56.576634 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:56.677482 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:56.677433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:56.733647 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:56.733609 2581 generic.go:356] "Generic (PLEG): container finished" podID="e21bca5a-adea-4fc4-865d-aa2d5b68d18a" containerID="8ce7e9779d81bd72b30d9e68826ada28d9904cc4c464941724784ddd274b4747" exitCode=0 Sep 10 14:48:56.734057 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:56.733690 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc59w" event={"ID":"e21bca5a-adea-4fc4-865d-aa2d5b68d18a","Type":"ContainerDied","Data":"8ce7e9779d81bd72b30d9e68826ada28d9904cc4c464941724784ddd274b4747"} Sep 10 14:48:56.754036 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:56.753999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:57.119923 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:57.119831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:57.739417 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:57.739378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc59w" event={"ID":"e21bca5a-adea-4fc4-865d-aa2d5b68d18a","Type":"ContainerStarted","Data":"9c9fb41388e93aafd6484668fb49f1341503f4217aeb3e5ed9f655116ebbffe3"} Sep 10 14:48:57.739923 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:57.739437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zc59w" event={"ID":"e21bca5a-adea-4fc4-865d-aa2d5b68d18a","Type":"ContainerStarted","Data":"967b7e056eb72a1fba469a87003331bdeaee96fd5230f01c427da822054cc890"} Sep 10 14:48:57.758714 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:57.758664 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zc59w" podStartSLOduration=3.5102066 podStartE2EDuration="4.75864724s" totalImagesPullingTime="1.17404104s" totalInitContainerRuntime="74.3996ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:57.757312385 +0000 UTC m=+92.051614513" watchObservedRunningTime="2026-09-10 14:48:57.75864724 +0000 UTC m=+92.052949369" Sep 10 14:48:58.438122 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:58.438087 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:58.539265 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:58.539230 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:58.680029 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:58.679991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:59.920121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:48:59.920082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:00.356244 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:00.356206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:00.455739 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:00.455693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:00.555142 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:00.555104 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:01.744601 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:01.744563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:01.807707 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:01.807654 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:02.366678 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:02.366637 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:02.464451 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:02.464403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:02.616364 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:02.616337 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jxmnh" Sep 10 14:49:03.318215 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:03.318180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:03.370685 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:03.370652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:04.986777 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:04.986730 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:05.087593 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.087550 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:05.322698 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.322615 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" podUID="8c4282dd-0e3a-407e-a5fa-923831d9684e" containerName="registry" containerID="cri-o://b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b" gracePeriod=30 Sep 10 14:49:05.566850 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.566793 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:49:05.682442 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682358 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682442 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682399 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682445 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682499 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n4mw\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682531 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682581 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.682663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682635 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets\") pod \"8c4282dd-0e3a-407e-a5fa-923831d9684e\" (UID: \"8c4282dd-0e3a-407e-a5fa-923831d9684e\") " Sep 10 14:49:05.683019 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.682974 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:05.683289 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.683241 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:05.685609 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.685568 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:05.685746 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.685714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:05.685843 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.685749 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:05.685904 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.685889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:05.685959 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.685928 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw" (OuterVolumeSpecName: "kube-api-access-2n4mw") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "kube-api-access-2n4mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:05.692537 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.692510 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8c4282dd-0e3a-407e-a5fa-923831d9684e" (UID: "8c4282dd-0e3a-407e-a5fa-923831d9684e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:49:05.770924 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.770889 2581 generic.go:356] "Generic (PLEG): container finished" podID="8c4282dd-0e3a-407e-a5fa-923831d9684e" containerID="b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b" exitCode=0 Sep 10 14:49:05.771090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.770936 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" Sep 10 14:49:05.771090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.770982 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" event={"ID":"8c4282dd-0e3a-407e-a5fa-923831d9684e","Type":"ContainerDied","Data":"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b"} Sep 10 14:49:05.771090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.771057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d57f78776-7wqpj" event={"ID":"8c4282dd-0e3a-407e-a5fa-923831d9684e","Type":"ContainerDied","Data":"39c3b5010b7dfee38b242716fd57397669e85db36613307aa35145db82b62794"} Sep 10 14:49:05.771090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.771077 2581 scope.go:117] "RemoveContainer" containerID="b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b" Sep 10 14:49:05.780329 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.780308 2581 scope.go:117] "RemoveContainer" containerID="b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b" Sep 10 14:49:05.780654 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:49:05.780634 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b\": container with ID starting with b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b not found: ID does not exist" containerID="b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b" Sep 10 14:49:05.780696 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.780665 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b"} err="failed to get container status \"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b\": rpc error: code = NotFound desc = could not find container \"b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b\": container with ID starting with b186fd5fa417c831e5ab67d10c1e774d80540ac597ebcc299e9b597e2211364b not found: ID does not exist" Sep 10 14:49:05.783163 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783147 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-trusted-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783166 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-image-registry-private-configuration\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783177 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-bound-sa-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783186 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4mw\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-kube-api-access-2n4mw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783195 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783203 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c4282dd-0e3a-407e-a5fa-923831d9684e-ca-trust-extracted\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783211 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c4282dd-0e3a-407e-a5fa-923831d9684e-registry-certificates\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.783221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.783219 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c4282dd-0e3a-407e-a5fa-923831d9684e-installation-pull-secrets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:49:05.790769 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.790738 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:49:05.796342 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:05.796320 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d57f78776-7wqpj"] Sep 10 14:49:06.228444 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:06.228414 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4282dd-0e3a-407e-a5fa-923831d9684e" path="/var/lib/kubelet/pods/8c4282dd-0e3a-407e-a5fa-923831d9684e/volumes" Sep 10 14:49:08.500178 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:08.500141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:08.600364 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:08.600326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:08.655615 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:08.655584 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:08.723034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:08.722992 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:08.824266 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:08.824177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:09.585000 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:09.584965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:09.707264 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:09.707228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:12.142582 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:12.142548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:12.197324 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:12.197288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:12.579808 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:12.579775 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:12.680498 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:12.680468 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:13.201976 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:13.201941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:15.859730 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:15.859692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:15.960644 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:15.960610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:22.840778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:22.836589 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" event={"ID":"e5a0946b-0765-4156-b0f8-6759a19ab380","Type":"ContainerDied","Data":"881323adb07c11e5f3abfad098878a58c721567f0ccff788cf682484c09ad461"} Sep 10 14:49:22.840778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:22.836606 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5a0946b-0765-4156-b0f8-6759a19ab380" containerID="881323adb07c11e5f3abfad098878a58c721567f0ccff788cf682484c09ad461" exitCode=0 Sep 10 14:49:22.840778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:22.837134 2581 scope.go:117] "RemoveContainer" containerID="881323adb07c11e5f3abfad098878a58c721567f0ccff788cf682484c09ad461" Sep 10 14:49:23.842388 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:23.842352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-5x6vj" event={"ID":"e5a0946b-0765-4156-b0f8-6759a19ab380","Type":"ContainerStarted","Data":"f712446a5f2209c4f7a2d4dcd226cee23c7b9d992998bc9fcb68e9cb34da262d"} Sep 10 14:49:30.445667 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:30.445630 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:30.504475 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:30.504446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:32.874345 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:32.874308 2581 generic.go:356] "Generic (PLEG): container finished" podID="b7d8884e-96c9-435b-8150-175a79139143" containerID="3ff5e794b1e8f244790051617ed6819c0f81c27ecbf45969fb5d7b116fd425bb" exitCode=0 Sep 10 14:49:32.874778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:32.874383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" event={"ID":"b7d8884e-96c9-435b-8150-175a79139143","Type":"ContainerDied","Data":"3ff5e794b1e8f244790051617ed6819c0f81c27ecbf45969fb5d7b116fd425bb"} Sep 10 14:49:32.874778 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:32.874704 2581 scope.go:117] "RemoveContainer" containerID="3ff5e794b1e8f244790051617ed6819c0f81c27ecbf45969fb5d7b116fd425bb" Sep 10 14:49:33.880617 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:33.880586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jhd4p" event={"ID":"b7d8884e-96c9-435b-8150-175a79139143","Type":"ContainerStarted","Data":"f9dc2a1db8558cb138f2a2665476ce34d5fa4e770a1678f9f20838767d0917a2"} Sep 10 14:49:35.016506 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.016463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:49:35.019028 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.019005 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/238df29e-1e13-4956-9f42-ed8340fd572f-metrics-certs\") pod \"network-metrics-daemon-bkhbf\" (UID: \"238df29e-1e13-4956-9f42-ed8340fd572f\") " pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:49:35.244726 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.244693 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\"" Sep 10 14:49:35.252761 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.252728 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bkhbf" Sep 10 14:49:35.520245 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.520174 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bkhbf"] Sep 10 14:49:35.522510 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:49:35.522471 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238df29e_1e13_4956_9f42_ed8340fd572f.slice/crio-7a1e34a3b29d9e79afeda09361c3131fb87ccd573dd1dc9bbd65357fce20dd02 WatchSource:0}: Error finding container 7a1e34a3b29d9e79afeda09361c3131fb87ccd573dd1dc9bbd65357fce20dd02: Status 404 returned error can't find the container with id 7a1e34a3b29d9e79afeda09361c3131fb87ccd573dd1dc9bbd65357fce20dd02 Sep 10 14:49:35.889485 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:35.889452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkhbf" event={"ID":"238df29e-1e13-4956-9f42-ed8340fd572f","Type":"ContainerStarted","Data":"7a1e34a3b29d9e79afeda09361c3131fb87ccd573dd1dc9bbd65357fce20dd02"} Sep 10 14:49:36.895482 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:36.895392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkhbf" event={"ID":"238df29e-1e13-4956-9f42-ed8340fd572f","Type":"ContainerStarted","Data":"bc6045e2373e6a00955cafe9aa84836c477c101bc8bef3948320f7bb7c878200"} Sep 10 14:49:36.895482 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:36.895473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bkhbf" event={"ID":"238df29e-1e13-4956-9f42-ed8340fd572f","Type":"ContainerStarted","Data":"960377ecd64ddf6dc5c5700047bc9f176f9fa8fa40724613ba82b06643d4ac72"} Sep 10 14:49:36.911887 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:36.911831 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bkhbf" podStartSLOduration=128.935114774 podStartE2EDuration="2m9.911814353s" totalImagesPullingTime="976.699579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:36.910839967 +0000 UTC m=+131.205142095" watchObservedRunningTime="2026-09-10 14:49:36.911814353 +0000 UTC m=+131.206116520" Sep 10 14:49:36.992936 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:36.992897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:37.046307 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:37.046260 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:37.491576 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:37.491521 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:37.593089 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:37.593059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:37.645409 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:37.645381 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:43.669090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:43.669046 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" podUID="a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 14:49:53.669220 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:49:53.669178 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" podUID="a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 14:50:03.669053 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.669008 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" podUID="a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 10 14:50:03.669508 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.669097 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" Sep 10 14:50:03.669568 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.669550 2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2f10f7e92ea6d4576b1654fb6c9cfd4d1966cb5a5fdc2483029edd0731400af9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 10 14:50:03.669607 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.669588 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" podUID="a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419" containerName="service-proxy" containerID="cri-o://2f10f7e92ea6d4576b1654fb6c9cfd4d1966cb5a5fdc2483029edd0731400af9" gracePeriod=30 Sep 10 14:50:03.998278 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.998242 2581 generic.go:356] "Generic (PLEG): container finished" podID="a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419" containerID="2f10f7e92ea6d4576b1654fb6c9cfd4d1966cb5a5fdc2483029edd0731400af9" exitCode=2 Sep 10 14:50:03.998437 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.998285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerDied","Data":"2f10f7e92ea6d4576b1654fb6c9cfd4d1966cb5a5fdc2483029edd0731400af9"} Sep 10 14:50:03.998437 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:03.998319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-668b6df74-fdnsx" event={"ID":"a0d9bfaa-2a71-4058-aa1f-eeea2c3e9419","Type":"ContainerStarted","Data":"0fbadcdad4380686d0ead474a1b48098ed484146fe26713272b16e64b3c1ab42"} Sep 10 14:50:12.847892 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:12.847844 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:13.052517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:13.052487 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:14.432112 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:14.432073 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:14.488428 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:14.488389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:21.837192 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:21.837101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:21.934064 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:21.934025 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:31.518270 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:31.518232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:32.870658 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:32.870619 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:32.923727 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:32.923690 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:32.986730 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:50:32.986694 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:28.089140 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:28.089084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:28.147292 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:28.147256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:36.269465 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:36.269425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:36.370321 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:36.370289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:40.129772 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:40.129728 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:40.187102 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:40.187049 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:45.456104 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:45.456056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:51:48.506458 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:51:48.506372 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:07.134891 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:07.134852 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:21.559269 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:21.559228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:21.650136 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:21.650101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:26.125575 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:26.125541 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:52:26.127868 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:26.127839 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:52:26.136274 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:26.136250 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 14:52:28.157804 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:28.157749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:28.259546 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:28.259502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:48.174394 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:48.174359 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:49.013829 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:49.013786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:51.793118 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:51.793081 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:51.892093 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:51.892052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:53.323445 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:53.323390 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:53.424048 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:53.424006 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:53.704456 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:53.704350 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:53.812326 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:53.812295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:55.158390 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:55.158350 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:55.711800 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:55.711739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:58.583428 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:58.583387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:52:58.682478 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:52:58.682446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:01.914452 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:01.914410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:04.515836 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:04.515798 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:04.619806 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:04.619745 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:04.717039 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:04.716999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:12.085672 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:12.085636 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:12.186046 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:12.186002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:14.007536 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:14.007495 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:25.781529 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:25.781440 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:25.881042 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:25.881004 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.354732 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.354688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.544312 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.544283 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.645024 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.644927 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.691674 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.691644 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9"] Sep 10 14:53:26.691952 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.691938 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4282dd-0e3a-407e-a5fa-923831d9684e" containerName="registry" Sep 10 14:53:26.692002 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.691954 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4282dd-0e3a-407e-a5fa-923831d9684e" containerName="registry" Sep 10 14:53:26.692036 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.692001 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4282dd-0e3a-407e-a5fa-923831d9684e" containerName="registry" Sep 10 14:53:26.694694 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.694678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.696906 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.696870 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 14:53:26.697042 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.696966 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-89wd6\"" Sep 10 14:53:26.697133 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.697113 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 14:53:26.697261 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.697248 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 14:53:26.697455 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.697440 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 14:53:26.708121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.708098 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9"] Sep 10 14:53:26.794475 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.794441 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.805422 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.805396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.805941 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.805438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.806133 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.806103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6f6v\" (UniqueName: \"kubernetes.io/projected/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-kube-api-access-n6f6v\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.806199 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.806166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.810537 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.806344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.894881 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.894828 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:26.907941 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.907872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.907941 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.907918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.908109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.907943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6f6v\" (UniqueName: \"kubernetes.io/projected/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-kube-api-access-n6f6v\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.908109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.907967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.908109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.907991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.908356 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.908329 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.910466 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.910441 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.910560 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.910530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.910681 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.910660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:26.919446 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:26.919422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6f6v\" (UniqueName: \"kubernetes.io/projected/720bcbe0-35b1-4f85-bdbb-44e3fa785d97-kube-api-access-n6f6v\") pod \"opendatahub-operator-controller-manager-759746987c-4b5g9\" (UID: \"720bcbe0-35b1-4f85-bdbb-44e3fa785d97\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:27.000424 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.000387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:27.005241 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.005217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:27.096249 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.096218 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:27.195046 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.194958 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:27.292314 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.292238 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9"] Sep 10 14:53:27.295097 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:53:27.295072 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720bcbe0_35b1_4f85_bdbb_44e3fa785d97.slice/crio-371468f6ebbf79129288f54df81a377fe599a4370b225ad89bac6ba46051c9cc WatchSource:0}: Error finding container 371468f6ebbf79129288f54df81a377fe599a4370b225ad89bac6ba46051c9cc: Status 404 returned error can't find the container with id 371468f6ebbf79129288f54df81a377fe599a4370b225ad89bac6ba46051c9cc Sep 10 14:53:27.296740 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.296720 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:53:27.754682 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:27.754646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" event={"ID":"720bcbe0-35b1-4f85-bdbb-44e3fa785d97","Type":"ContainerStarted","Data":"371468f6ebbf79129288f54df81a377fe599a4370b225ad89bac6ba46051c9cc"} Sep 10 14:53:30.771548 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:30.771504 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" event={"ID":"720bcbe0-35b1-4f85-bdbb-44e3fa785d97","Type":"ContainerStarted","Data":"a3c3cbbb14b58809346c576b1a3f0609bd0b7cef988e0210cb2717c619f1d9f4"} Sep 10 14:53:31.776595 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:31.776561 2581 generic.go:356] "Generic (PLEG): container finished" podID="720bcbe0-35b1-4f85-bdbb-44e3fa785d97" containerID="a3c3cbbb14b58809346c576b1a3f0609bd0b7cef988e0210cb2717c619f1d9f4" exitCode=0 Sep 10 14:53:31.777077 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:31.776605 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" event={"ID":"720bcbe0-35b1-4f85-bdbb-44e3fa785d97","Type":"ContainerDied","Data":"a3c3cbbb14b58809346c576b1a3f0609bd0b7cef988e0210cb2717c619f1d9f4"} Sep 10 14:53:32.783028 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:32.782986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" event={"ID":"720bcbe0-35b1-4f85-bdbb-44e3fa785d97","Type":"ContainerStarted","Data":"88c637c322e213f609451b10d944819fa9dea7a70b5a8f05b54a543fc0ce5a64"} Sep 10 14:53:32.783410 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:32.783120 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:32.808291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:32.808229 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" podStartSLOduration=2.431871205 podStartE2EDuration="6.808207866s" totalImagesPullingTime="3.364115781s" totalInitContainerRuntime="1.01222088s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:32.806479091 +0000 UTC m=+367.100781220" watchObservedRunningTime="2026-09-10 14:53:32.808207866 +0000 UTC m=+367.102509995" Sep 10 14:53:43.789657 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:43.789624 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4b5g9" Sep 10 14:53:46.163505 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.163467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:46.269451 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.269410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:46.504245 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.504204 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:46.553841 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.553813 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-7djdw"] Sep 10 14:53:46.563600 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.563575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.566722 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.566695 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 14:53:46.566956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.566897 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 14:53:46.566956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.566950 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 14:53:46.567082 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.566950 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 14:53:46.575581 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.575553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-7djdw"] Sep 10 14:53:46.659902 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.659868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.660071 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.659908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtlh\" (UniqueName: \"kubernetes.io/projected/060b937a-f5c4-4360-82ab-2842f5691632-kube-api-access-smtlh\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.660071 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.659933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/060b937a-f5c4-4360-82ab-2842f5691632-tmp\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.754380 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.754254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:46.761111 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.761084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.761304 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.761123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtlh\" (UniqueName: \"kubernetes.io/projected/060b937a-f5c4-4360-82ab-2842f5691632-kube-api-access-smtlh\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.761304 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:53:46.761230 2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 10 14:53:46.761304 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:53:46.761298 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs podName:060b937a-f5c4-4360-82ab-2842f5691632 nodeName:}" failed. No retries permitted until 2026-09-10 14:53:47.261276965 +0000 UTC m=+381.555579075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs") pod "kube-auth-proxy-7497c74497-7djdw" (UID: "060b937a-f5c4-4360-82ab-2842f5691632") : secret "kube-auth-proxy-tls" not found Sep 10 14:53:46.761472 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.761443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/060b937a-f5c4-4360-82ab-2842f5691632-tmp\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.763811 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.763787 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/060b937a-f5c4-4360-82ab-2842f5691632-tmp\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.779559 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.779536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtlh\" (UniqueName: \"kubernetes.io/projected/060b937a-f5c4-4360-82ab-2842f5691632-kube-api-access-smtlh\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:46.847297 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.847262 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:46.928612 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:46.928570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:47.266589 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:47.266542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:47.269981 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:47.269952 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060b937a-f5c4-4360-82ab-2842f5691632-tls-certs\") pod \"kube-auth-proxy-7497c74497-7djdw\" (UID: \"060b937a-f5c4-4360-82ab-2842f5691632\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:47.472730 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:47.472691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" Sep 10 14:53:47.755695 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:47.755662 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-7djdw"] Sep 10 14:53:47.758592 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:53:47.758559 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060b937a_f5c4_4360_82ab_2842f5691632.slice/crio-c3b1d367ad2bb3c0697e7a9b6c90e9525911b575a42422e800b40b1347c3264e WatchSource:0}: Error finding container c3b1d367ad2bb3c0697e7a9b6c90e9525911b575a42422e800b40b1347c3264e: Status 404 returned error can't find the container with id c3b1d367ad2bb3c0697e7a9b6c90e9525911b575a42422e800b40b1347c3264e Sep 10 14:53:47.841519 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:47.841441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" event={"ID":"060b937a-f5c4-4360-82ab-2842f5691632","Type":"ContainerStarted","Data":"c3b1d367ad2bb3c0697e7a9b6c90e9525911b575a42422e800b40b1347c3264e"} Sep 10 14:53:48.027142 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.027102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:48.027322 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.027308 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:53:48.029922 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.029895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.032144 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.032121 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 14:53:48.032144 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.032138 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 14:53:48.032327 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.032120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 14:53:48.032327 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.032196 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 14:53:48.032327 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.032211 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c7gxb\"" Sep 10 14:53:48.045433 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.045404 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:53:48.087280 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.087246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:48.174611 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174611 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174611 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174667 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5244\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.174908 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.174782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275316 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5244\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.275820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.275538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.276637 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.276581 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.279541 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.279511 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.279988 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.279902 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.279988 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.279962 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.280253 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.280234 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.285105 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.285057 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5244\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.285667 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.285648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6gc46\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.339619 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.339584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:48.631889 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.631720 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:53:48.634827 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:53:48.634796 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6e8a8a_aae8_43c9_b552_b33bcb9589fb.slice/crio-936998955f79b2c4422f631b4ecc1b5a0ac88bbe0dd4914349c5c40a9005dcb7 WatchSource:0}: Error finding container 936998955f79b2c4422f631b4ecc1b5a0ac88bbe0dd4914349c5c40a9005dcb7: Status 404 returned error can't find the container with id 936998955f79b2c4422f631b4ecc1b5a0ac88bbe0dd4914349c5c40a9005dcb7 Sep 10 14:53:48.848020 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:48.847976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" event={"ID":"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb","Type":"ContainerStarted","Data":"936998955f79b2c4422f631b4ecc1b5a0ac88bbe0dd4914349c5c40a9005dcb7"} Sep 10 14:53:49.001939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:49.001891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:50.857488 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:50.857446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" event={"ID":"060b937a-f5c4-4360-82ab-2842f5691632","Type":"ContainerStarted","Data":"c869ff28ab26b48021013a228a4f69768ff841443f404c183ba553026da52fe2"} Sep 10 14:53:50.874206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:50.874150 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7497c74497-7djdw" podStartSLOduration=2.345330058 podStartE2EDuration="4.874133656s" totalImagesPullingTime="2.528803598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:50.873052609 +0000 UTC m=+385.167354739" watchObservedRunningTime="2026-09-10 14:53:50.874133656 +0000 UTC m=+385.168435785" Sep 10 14:53:52.588251 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:52.588209 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 10 14:53:52.588486 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:52.588287 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 10 14:53:52.868002 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:52.867912 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" event={"ID":"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb","Type":"ContainerStarted","Data":"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab"} Sep 10 14:53:52.868166 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:52.868052 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:52.891219 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:52.891173 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" podStartSLOduration=0.940904305 podStartE2EDuration="4.891158888s" totalImagesPullingTime="3.950254583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:52.890660976 +0000 UTC m=+387.184963105" watchObservedRunningTime="2026-09-10 14:53:52.891158888 +0000 UTC m=+387.185461018" Sep 10 14:53:53.078034 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.077945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:53.177126 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.177011 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:53.417073 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.417044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:53.521303 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.521244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:53.617649 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.617617 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:53.874091 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:53.874011 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:53:57.296943 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.296904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:57.371603 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.371566 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:57.439426 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.439392 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:57.535061 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.535022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:57.818590 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.818554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:57.927612 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:57.927564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:58.324863 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:53:58.324825 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:04.082942 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:04.082903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.459574 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.459538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.514707 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.514670 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf"] Sep 10 14:54:06.517641 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.517623 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.520514 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.520497 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 14:54:06.520614 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.520515 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w96f8\"" Sep 10 14:54:06.535269 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.535244 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5"] Sep 10 14:54:06.537862 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.537840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.537962 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.537931 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf"] Sep 10 14:54:06.553642 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.553610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.555784 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.555748 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 14:54:06.555876 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.555849 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jhglp\"" Sep 10 14:54:06.557292 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.557273 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5"] Sep 10 14:54:06.629377 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.629343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrl4l\" (UniqueName: \"kubernetes.io/projected/0664a5c7-233f-4dbd-9c8b-189954ce8391-kube-api-access-qrl4l\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.629572 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.629385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0664a5c7-233f-4dbd-9c8b-189954ce8391-cert\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.629572 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.629533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.629684 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.629600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhmw\" (UniqueName: \"kubernetes.io/projected/6c13eac1-9db5-48b3-8ed7-1425b0949669-kube-api-access-pwhmw\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.731012 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.730907 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.731012 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.730981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhmw\" (UniqueName: \"kubernetes.io/projected/6c13eac1-9db5-48b3-8ed7-1425b0949669-kube-api-access-pwhmw\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.731247 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.731020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrl4l\" (UniqueName: \"kubernetes.io/projected/0664a5c7-233f-4dbd-9c8b-189954ce8391-kube-api-access-qrl4l\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.731247 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.731049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0664a5c7-233f-4dbd-9c8b-189954ce8391-cert\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.731247 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:54:06.731065 2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 14:54:06.731247 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:54:06.731138 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert podName:6c13eac1-9db5-48b3-8ed7-1425b0949669 nodeName:}" failed. No retries permitted until 2026-09-10 14:54:07.231117489 +0000 UTC m=+401.525419606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert") pod "llmisvc-controller-manager-777c6bd7d5-hbvz5" (UID: "6c13eac1-9db5-48b3-8ed7-1425b0949669") : secret "llmisvc-webhook-server-cert" not found Sep 10 14:54:06.733867 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.733841 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0664a5c7-233f-4dbd-9c8b-189954ce8391-cert\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.740589 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.740566 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrl4l\" (UniqueName: \"kubernetes.io/projected/0664a5c7-233f-4dbd-9c8b-189954ce8391-kube-api-access-qrl4l\") pod \"kserve-controller-manager-d6f4b65b8-v4fhf\" (UID: \"0664a5c7-233f-4dbd-9c8b-189954ce8391\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.741261 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.741232 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhmw\" (UniqueName: \"kubernetes.io/projected/6c13eac1-9db5-48b3-8ed7-1425b0949669-kube-api-access-pwhmw\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:06.772191 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.772155 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.827237 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.827199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:06.872887 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:06.872840 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:07.098025 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.097992 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf"] Sep 10 14:54:07.100406 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:54:07.100379 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0664a5c7_233f_4dbd_9c8b_189954ce8391.slice/crio-3028aaea282a4ad5230c945b1763d1b1b412f0d5e5783003a9d2648a68d92b6e WatchSource:0}: Error finding container 3028aaea282a4ad5230c945b1763d1b1b412f0d5e5783003a9d2648a68d92b6e: Status 404 returned error can't find the container with id 3028aaea282a4ad5230c945b1763d1b1b412f0d5e5783003a9d2648a68d92b6e Sep 10 14:54:07.235636 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.235590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:07.238246 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.238224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c13eac1-9db5-48b3-8ed7-1425b0949669-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbvz5\" (UID: \"6c13eac1-9db5-48b3-8ed7-1425b0949669\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:07.448212 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.448122 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:07.714738 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.714645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:07.719155 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.719131 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5"] Sep 10 14:54:07.722685 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:54:07.722657 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6c13eac1_9db5_48b3_8ed7_1425b0949669.slice/crio-b02c03dcccce6c6e81a599c9db1182ec70321ca9b212a2c4d958c8c173a77dc1 WatchSource:0}: Error finding container b02c03dcccce6c6e81a599c9db1182ec70321ca9b212a2c4d958c8c173a77dc1: Status 404 returned error can't find the container with id b02c03dcccce6c6e81a599c9db1182ec70321ca9b212a2c4d958c8c173a77dc1 Sep 10 14:54:07.814535 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.814500 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:07.870077 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.870044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:07.931777 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.931712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" event={"ID":"6c13eac1-9db5-48b3-8ed7-1425b0949669","Type":"ContainerStarted","Data":"b02c03dcccce6c6e81a599c9db1182ec70321ca9b212a2c4d958c8c173a77dc1"} Sep 10 14:54:07.933220 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:07.933188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" event={"ID":"0664a5c7-233f-4dbd-9c8b-189954ce8391","Type":"ContainerStarted","Data":"3028aaea282a4ad5230c945b1763d1b1b412f0d5e5783003a9d2648a68d92b6e"} Sep 10 14:54:08.516833 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:08.516796 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:54:08.517071 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:08.517044 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerName="discovery" containerID="cri-o://78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab" gracePeriod=30 Sep 10 14:54:08.873114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:08.873032 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6gc46 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.22:8080/ready\": dial tcp 10.132.0.22:8080: connect: connection refused" start-of-body= Sep 10 14:54:08.873114 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:08.873096 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.22:8080/ready\": dial tcp 10.132.0.22:8080: connect: connection refused" Sep 10 14:54:09.340793 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.340742 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:54:09.456385 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456279 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5244\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456703 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456610 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456703 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456690 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456894 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456894 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456894 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456849 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.456894 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.456873 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert\") pod \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\" (UID: \"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb\") " Sep 10 14:54:09.457498 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.457450 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:54:09.459901 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.459841 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:54:09.460009 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.459938 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:54:09.460234 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.460209 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts" (OuterVolumeSpecName: "cacerts") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:54:09.460341 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.460319 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:54:09.460431 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.460390 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244" (OuterVolumeSpecName: "kube-api-access-h5244") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "kube-api-access-h5244". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:54:09.461641 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.461623 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs" (OuterVolumeSpecName: "local-certs") pod "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" (UID: "1f6e8a8a-aae8-43c9-b552-b33bcb9589fb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:54:09.557852 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557816 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5244\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-kube-api-access-h5244\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.557852 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557847 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-kubeconfig\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.557852 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557856 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-cacerts\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.558128 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557865 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-local-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.558128 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557874 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-ca-configmap\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.558128 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557881 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.558128 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.557889 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb-istio-csr-dns-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:54:09.944429 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.944386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" event={"ID":"0664a5c7-233f-4dbd-9c8b-189954ce8391","Type":"ContainerStarted","Data":"f1a00affa9e4e3c6ee70b7d6ad6d59e97fc3b9b74781d21355a4ef058da7851e"} Sep 10 14:54:09.944938 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.944509 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:09.946080 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.945991 2581 generic.go:356] "Generic (PLEG): container finished" podID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerID="78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab" exitCode=0 Sep 10 14:54:09.946080 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.946050 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" Sep 10 14:54:09.946259 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.946083 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" event={"ID":"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb","Type":"ContainerDied","Data":"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab"} Sep 10 14:54:09.946259 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.946141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46" event={"ID":"1f6e8a8a-aae8-43c9-b552-b33bcb9589fb","Type":"ContainerDied","Data":"936998955f79b2c4422f631b4ecc1b5a0ac88bbe0dd4914349c5c40a9005dcb7"} Sep 10 14:54:09.946259 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.946165 2581 scope.go:117] "RemoveContainer" containerID="78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab" Sep 10 14:54:09.957566 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.957531 2581 scope.go:117] "RemoveContainer" containerID="78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab" Sep 10 14:54:09.957992 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:54:09.957967 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab\": container with ID starting with 78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab not found: ID does not exist" containerID="78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab" Sep 10 14:54:09.958084 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.958005 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab"} err="failed to get container status \"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab\": rpc error: code = NotFound desc = could not find container \"78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab\": container with ID starting with 78f3211318b220d30c2aad9a8ecd43120a3ae1ee42d424d49e5a271cf74200ab not found: ID does not exist" Sep 10 14:54:09.962054 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.962010 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" podStartSLOduration=1.703647503 podStartE2EDuration="3.961992356s" totalImagesPullingTime="2.258344853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:09.959934121 +0000 UTC m=+404.254236253" watchObservedRunningTime="2026-09-10 14:54:09.961992356 +0000 UTC m=+404.256294486" Sep 10 14:54:09.975518 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.975487 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:54:09.982059 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:09.982031 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6gc46"] Sep 10 14:54:10.229436 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:10.229356 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" path="/var/lib/kubelet/pods/1f6e8a8a-aae8-43c9-b552-b33bcb9589fb/volumes" Sep 10 14:54:11.958688 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:11.958657 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" event={"ID":"6c13eac1-9db5-48b3-8ed7-1425b0949669","Type":"ContainerStarted","Data":"6ce7512372807ae3c0ad8ed87fae27d7682f264fb5792e601d45feadac3f5734"} Sep 10 14:54:11.959109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:11.958717 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:11.975522 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:11.975478 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" podStartSLOduration=2.630096761 podStartE2EDuration="5.975464949s" totalImagesPullingTime="3.345368188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:11.974642941 +0000 UTC m=+406.268945070" watchObservedRunningTime="2026-09-10 14:54:11.975464949 +0000 UTC m=+406.269767078" Sep 10 14:54:40.958479 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:40.958443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:54:42.965845 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:42.965812 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbvz5" Sep 10 14:54:58.233308 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:58.233275 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:58.335711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:54:58.335672 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:13.384679 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:13.384644 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:14.189918 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:14.189880 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:14.982097 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:14.982065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:16.230037 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:16.230006 2581 generic.go:356] "Generic (PLEG): container finished" podID="0664a5c7-233f-4dbd-9c8b-189954ce8391" containerID="f1a00affa9e4e3c6ee70b7d6ad6d59e97fc3b9b74781d21355a4ef058da7851e" exitCode=1 Sep 10 14:55:16.230482 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:16.230083 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" event={"ID":"0664a5c7-233f-4dbd-9c8b-189954ce8391","Type":"ContainerDied","Data":"f1a00affa9e4e3c6ee70b7d6ad6d59e97fc3b9b74781d21355a4ef058da7851e"} Sep 10 14:55:16.230550 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:16.230481 2581 scope.go:117] "RemoveContainer" containerID="f1a00affa9e4e3c6ee70b7d6ad6d59e97fc3b9b74781d21355a4ef058da7851e" Sep 10 14:55:16.827879 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:16.827781 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:55:17.235663 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.235578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" event={"ID":"0664a5c7-233f-4dbd-9c8b-189954ce8391","Type":"ContainerStarted","Data":"71dfa3cce63c888a87a921d0ed0dffc2e28227c9cf27bf6bfec0f243b274667b"} Sep 10 14:55:17.236035 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.235687 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:55:17.737406 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.737372 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks"] Sep 10 14:55:17.737678 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.737666 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerName="discovery" Sep 10 14:55:17.737721 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.737680 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerName="discovery" Sep 10 14:55:17.737780 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.737735 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6e8a8a-aae8-43c9-b552-b33bcb9589fb" containerName="discovery" Sep 10 14:55:17.739720 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.739703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:17.742073 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.742046 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 14:55:17.742192 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.742078 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 14:55:17.742192 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.742156 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hwqjk\"" Sep 10 14:55:17.752087 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.752068 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks"] Sep 10 14:55:17.888681 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.888643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:17.888681 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.888687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zvc\" (UniqueName: \"kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:17.990024 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.989929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:17.990024 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.989975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zvc\" (UniqueName: \"kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:17.990330 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:17.990310 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:18.001157 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.001132 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zvc\" (UniqueName: \"kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:18.049993 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.049943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:18.275732 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.275646 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:18.323552 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.323519 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks"] Sep 10 14:55:18.326802 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:18.326774 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5579ed59_05e1_4b50_84ca_d3c312c642d2.slice/crio-f24ae6501f24e44281c3ad57448996f89d2a8493cbb2c3933dc3c35f76c06d67 WatchSource:0}: Error finding container f24ae6501f24e44281c3ad57448996f89d2a8493cbb2c3933dc3c35f76c06d67: Status 404 returned error can't find the container with id f24ae6501f24e44281c3ad57448996f89d2a8493cbb2c3933dc3c35f76c06d67 Sep 10 14:55:18.373982 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.373948 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:18.476306 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:18.476274 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:19.245404 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.245325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" event={"ID":"5579ed59-05e1-4b50-84ca-d3c312c642d2","Type":"ContainerStarted","Data":"f24ae6501f24e44281c3ad57448996f89d2a8493cbb2c3933dc3c35f76c06d67"} Sep 10 14:55:19.691191 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.691091 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m"] Sep 10 14:55:19.693835 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.693806 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:19.696395 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.696368 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-scnpp\"" Sep 10 14:55:19.696658 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.696637 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 14:55:19.704911 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.704887 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m"] Sep 10 14:55:19.806117 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.806076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9vc\" (UniqueName: \"kubernetes.io/projected/bf78bd35-1d12-42de-8357-bdc308efbe69-kube-api-access-mz9vc\") pod \"dns-operator-controller-manager-7b84cd7c9c-w5m4m\" (UID: \"bf78bd35-1d12-42de-8357-bdc308efbe69\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:19.906811 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.906711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9vc\" (UniqueName: \"kubernetes.io/projected/bf78bd35-1d12-42de-8357-bdc308efbe69-kube-api-access-mz9vc\") pod \"dns-operator-controller-manager-7b84cd7c9c-w5m4m\" (UID: \"bf78bd35-1d12-42de-8357-bdc308efbe69\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:19.917109 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:19.917048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9vc\" (UniqueName: \"kubernetes.io/projected/bf78bd35-1d12-42de-8357-bdc308efbe69-kube-api-access-mz9vc\") pod \"dns-operator-controller-manager-7b84cd7c9c-w5m4m\" (UID: \"bf78bd35-1d12-42de-8357-bdc308efbe69\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:20.008735 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:20.008698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:20.430146 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:20.430015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:20.637469 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:20.637430 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:21.058889 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:21.058794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m"] Sep 10 14:55:21.063700 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:21.063665 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78bd35_1d12_42de_8357_bdc308efbe69.slice/crio-c54b1084d6dc6e2b877fb028dbaf0de6e5592f5a47fb81a6f4e09f818ddf6c7c WatchSource:0}: Error finding container c54b1084d6dc6e2b877fb028dbaf0de6e5592f5a47fb81a6f4e09f818ddf6c7c: Status 404 returned error can't find the container with id c54b1084d6dc6e2b877fb028dbaf0de6e5592f5a47fb81a6f4e09f818ddf6c7c Sep 10 14:55:21.259323 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:21.259281 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" event={"ID":"bf78bd35-1d12-42de-8357-bdc308efbe69","Type":"ContainerStarted","Data":"c54b1084d6dc6e2b877fb028dbaf0de6e5592f5a47fb81a6f4e09f818ddf6c7c"} Sep 10 14:55:22.143887 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.143852 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x"] Sep 10 14:55:22.146727 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.146698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:22.149483 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.149438 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lprtt\"" Sep 10 14:55:22.160883 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.160843 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x"] Sep 10 14:55:22.191120 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.191063 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:22.291123 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.291087 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:22.329231 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.329163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdrk\" (UniqueName: \"kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk\") pod \"limitador-operator-controller-manager-74495c69f8-r4l2x\" (UID: \"a878fb2d-527a-4969-b7ff-3c3ece425121\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:22.430378 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.430345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdrk\" (UniqueName: \"kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk\") pod \"limitador-operator-controller-manager-74495c69f8-r4l2x\" (UID: \"a878fb2d-527a-4969-b7ff-3c3ece425121\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:22.440032 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.439995 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdrk\" (UniqueName: \"kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk\") pod \"limitador-operator-controller-manager-74495c69f8-r4l2x\" (UID: \"a878fb2d-527a-4969-b7ff-3c3ece425121\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:22.460597 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.460566 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:22.742942 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.742906 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x"] Sep 10 14:55:22.745668 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:22.745637 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda878fb2d_527a_4969_b7ff_3c3ece425121.slice/crio-5759c352c1fe9c3eab1ab232da9a0c302873e639d5c6e2281b3069df45bce3e6 WatchSource:0}: Error finding container 5759c352c1fe9c3eab1ab232da9a0c302873e639d5c6e2281b3069df45bce3e6: Status 404 returned error can't find the container with id 5759c352c1fe9c3eab1ab232da9a0c302873e639d5c6e2281b3069df45bce3e6 Sep 10 14:55:22.795453 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:22.795420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:23.269926 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:23.269894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" event={"ID":"a878fb2d-527a-4969-b7ff-3c3ece425121","Type":"ContainerStarted","Data":"5759c352c1fe9c3eab1ab232da9a0c302873e639d5c6e2281b3069df45bce3e6"} Sep 10 14:55:23.271708 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:23.271683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" event={"ID":"5579ed59-05e1-4b50-84ca-d3c312c642d2","Type":"ContainerStarted","Data":"709ff1c8b29252080df2d807735fb4c1cdfd72938ed47730a0f87d15ce20257e"} Sep 10 14:55:23.271898 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:23.271880 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:23.302708 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:23.302587 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" podStartSLOduration=2.242234749 podStartE2EDuration="6.302564772s" totalImagesPullingTime="4.060330023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:23.300459027 +0000 UTC m=+477.594761162" watchObservedRunningTime="2026-09-10 14:55:23.302564772 +0000 UTC m=+477.596866903" Sep 10 14:55:24.238961 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.238927 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:24.283924 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.283884 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" event={"ID":"bf78bd35-1d12-42de-8357-bdc308efbe69","Type":"ContainerStarted","Data":"baf552300704cba4807e954ec294c31128f122cce52c9af08c31bfb43cb31dea"} Sep 10 14:55:24.284360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.284068 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:24.302984 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.302927 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" podStartSLOduration=3.111988888 podStartE2EDuration="5.302909886s" totalImagesPullingTime="2.190920998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:24.301771603 +0000 UTC m=+478.596073723" watchObservedRunningTime="2026-09-10 14:55:24.302909886 +0000 UTC m=+478.597212014" Sep 10 14:55:24.681810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.681778 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rhn68"] Sep 10 14:55:24.684029 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.684014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:24.686422 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.686392 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qbdsx\"" Sep 10 14:55:24.703585 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.703481 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rhn68"] Sep 10 14:55:24.852171 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.852080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqtz8\" (UniqueName: \"kubernetes.io/projected/4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6-kube-api-access-pqtz8\") pod \"authorino-operator-76b478bd78-rhn68\" (UID: \"4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:24.952973 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.952927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqtz8\" (UniqueName: \"kubernetes.io/projected/4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6-kube-api-access-pqtz8\") pod \"authorino-operator-76b478bd78-rhn68\" (UID: \"4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:24.961032 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.961004 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqtz8\" (UniqueName: \"kubernetes.io/projected/4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6-kube-api-access-pqtz8\") pod \"authorino-operator-76b478bd78-rhn68\" (UID: \"4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:24.993871 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:24.993834 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:25.276193 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.276156 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rhn68"] Sep 10 14:55:25.281714 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:25.281680 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1c3d32_4d01_4d9f_b97a_dfaa5c1cf9b6.slice/crio-fdf07f164ad601873a164618491eb8104d937977c117376388a547cf7d7bd9b8 WatchSource:0}: Error finding container fdf07f164ad601873a164618491eb8104d937977c117376388a547cf7d7bd9b8: Status 404 returned error can't find the container with id fdf07f164ad601873a164618491eb8104d937977c117376388a547cf7d7bd9b8 Sep 10 14:55:25.292925 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.292889 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" event={"ID":"a878fb2d-527a-4969-b7ff-3c3ece425121","Type":"ContainerStarted","Data":"d95980ac46c2a17c88be802f0cdc2d7f8dd86a617e3c6e6034c1d31957b0f6e7"} Sep 10 14:55:25.293342 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.293039 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:25.294488 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.294465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" event={"ID":"4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6","Type":"ContainerStarted","Data":"fdf07f164ad601873a164618491eb8104d937977c117376388a547cf7d7bd9b8"} Sep 10 14:55:25.330433 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.330360 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" podStartSLOduration=1.502719719 podStartE2EDuration="3.33034084s" totalImagesPullingTime="1.827621121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:25.328924009 +0000 UTC m=+479.623226175" watchObservedRunningTime="2026-09-10 14:55:25.33034084 +0000 UTC m=+479.624642972" Sep 10 14:55:25.548069 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.547989 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg"] Sep 10 14:55:25.550542 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.550517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:25.550840 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.550824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.553286 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.553266 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cm4wp\"" Sep 10 14:55:25.553534 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.553483 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 10 14:55:25.553861 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.553841 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 10 14:55:25.561825 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.561804 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg"] Sep 10 14:55:25.657711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.657674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/78066a1e-4aab-4b99-ab4b-ac180cf1514c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.657938 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.657731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78066a1e-4aab-4b99-ab4b-ac180cf1514c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.657938 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.657873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s9c\" (UniqueName: \"kubernetes.io/projected/78066a1e-4aab-4b99-ab4b-ac180cf1514c-kube-api-access-58s9c\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.687068 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.687032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:25.758617 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.758574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78066a1e-4aab-4b99-ab4b-ac180cf1514c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.758843 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.758660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s9c\" (UniqueName: \"kubernetes.io/projected/78066a1e-4aab-4b99-ab4b-ac180cf1514c-kube-api-access-58s9c\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.758843 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.758693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/78066a1e-4aab-4b99-ab4b-ac180cf1514c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.759375 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.759352 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78066a1e-4aab-4b99-ab4b-ac180cf1514c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.761310 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.761285 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/78066a1e-4aab-4b99-ab4b-ac180cf1514c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.766862 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.766829 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s9c\" (UniqueName: \"kubernetes.io/projected/78066a1e-4aab-4b99-ab4b-ac180cf1514c-kube-api-access-58s9c\") pod \"kuadrant-console-plugin-fbdd4f445-2p5vg\" (UID: \"78066a1e-4aab-4b99-ab4b-ac180cf1514c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:25.859799 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:25.859676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" Sep 10 14:55:26.137377 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:26.137352 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg"] Sep 10 14:55:26.140398 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:26.140369 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78066a1e_4aab_4b99_ab4b_ac180cf1514c.slice/crio-f755911aa3c1863eb1bc040de2dd281ad9f9fffd2a0b3a6fdf0a5f79ee233483 WatchSource:0}: Error finding container f755911aa3c1863eb1bc040de2dd281ad9f9fffd2a0b3a6fdf0a5f79ee233483: Status 404 returned error can't find the container with id f755911aa3c1863eb1bc040de2dd281ad9f9fffd2a0b3a6fdf0a5f79ee233483 Sep 10 14:55:26.299573 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:26.299535 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" event={"ID":"78066a1e-4aab-4b99-ab4b-ac180cf1514c","Type":"ContainerStarted","Data":"f755911aa3c1863eb1bc040de2dd281ad9f9fffd2a0b3a6fdf0a5f79ee233483"} Sep 10 14:55:28.312730 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:28.312678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" event={"ID":"4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6","Type":"ContainerStarted","Data":"ce977615757aa4d8305a434f469bed55ff385b0ef6412d9653ab8dcfdd6b3a7f"} Sep 10 14:55:28.313152 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:28.312773 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:28.330254 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:28.330184 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" podStartSLOduration=1.961243154 podStartE2EDuration="4.330159823s" totalImagesPullingTime="2.368916669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:28.328058384 +0000 UTC m=+482.622360513" watchObservedRunningTime="2026-09-10 14:55:28.330159823 +0000 UTC m=+482.624461954" Sep 10 14:55:28.594005 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:28.593923 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:28.645286 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:28.645256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:29.001884 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:29.001849 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:29.059949 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:29.059913 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:29.393301 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:29.393213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:29.523374 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:29.523268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:30.425216 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:30.425179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:30.486549 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:30.486515 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:31.329155 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:31.329124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" event={"ID":"78066a1e-4aab-4b99-ab4b-ac180cf1514c","Type":"ContainerStarted","Data":"291f9b15bc4e425aba820d8debb5c5cdc4c53dd8a6542fb4a3f8398d0fd5b14a"} Sep 10 14:55:31.344861 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:31.344805 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-2p5vg" podStartSLOduration=1.871334755 podStartE2EDuration="6.344787859s" totalImagesPullingTime="4.473453104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:31.343406778 +0000 UTC m=+485.637708907" watchObservedRunningTime="2026-09-10 14:55:31.344787859 +0000 UTC m=+485.639089988" Sep 10 14:55:32.645440 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:32.645403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:34.286389 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:34.286357 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:35.132291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:35.132250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:35.229002 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:35.228963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:35.296902 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:35.296872 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w5m4m" Sep 10 14:55:36.302588 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:36.302557 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:39.321225 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:39.321198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-rhn68" Sep 10 14:55:39.844518 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:39.844486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:39.963508 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:39.963472 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:44.606414 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:44.606382 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:44.718834 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:44.718800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:44.773661 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:44.773623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:45.121772 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.121732 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6"] Sep 10 14:55:45.127956 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.127929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.137744 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.137717 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6"] Sep 10 14:55:45.210105 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.210061 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls49k\" (UniqueName: \"kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.210279 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.210118 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.311459 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.311415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls49k\" (UniqueName: \"kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.311660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.311467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.311955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.311931 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.320255 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.320229 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls49k\" (UniqueName: \"kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.438325 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.438228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:45.723687 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:45.723664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6"] Sep 10 14:55:45.725311 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:45.725284 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdea041_d0d1_4241_b57f_63f4db525bc1.slice/crio-ca3151d360389abacfe4a23e7a6092c48d5826f4e4f52f957238948a7a6fd40a WatchSource:0}: Error finding container ca3151d360389abacfe4a23e7a6092c48d5826f4e4f52f957238948a7a6fd40a: Status 404 returned error can't find the container with id ca3151d360389abacfe4a23e7a6092c48d5826f4e4f52f957238948a7a6fd40a Sep 10 14:55:46.400086 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:46.400045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" event={"ID":"9bdea041-d0d1-4241-b57f-63f4db525bc1","Type":"ContainerStarted","Data":"583baba0f6c960be01c3767ec7c0203d4eaa513347219a8da351ef1f20a7c386"} Sep 10 14:55:46.400283 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:46.400101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" event={"ID":"9bdea041-d0d1-4241-b57f-63f4db525bc1","Type":"ContainerStarted","Data":"ca3151d360389abacfe4a23e7a6092c48d5826f4e4f52f957238948a7a6fd40a"} Sep 10 14:55:46.400283 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:46.400168 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:46.427424 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:46.427371 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" podStartSLOduration=1.427353711 podStartE2EDuration="1.427353711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:46.424992722 +0000 UTC m=+500.719294851" watchObservedRunningTime="2026-09-10 14:55:46.427353711 +0000 UTC m=+500.721655841" Sep 10 14:55:48.244643 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:48.244611 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v4fhf" Sep 10 14:55:49.818038 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:49.817976 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:49.920150 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:49.920112 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:50.215709 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:50.215601 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:50.318909 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:50.318862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:50.416241 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:50.416210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:50.516294 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:50.516258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:56.285559 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.285524 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6"] Sep 10 14:55:56.286090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.285715 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" containerName="manager" containerID="cri-o://583baba0f6c960be01c3767ec7c0203d4eaa513347219a8da351ef1f20a7c386" gracePeriod=2 Sep 10 14:55:56.287607 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.287580 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.290276 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.290250 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6"] Sep 10 14:55:56.295155 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.295115 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.306546 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.306518 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks"] Sep 10 14:55:56.306831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.306773 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" containerName="manager" containerID="cri-o://709ff1c8b29252080df2d807735fb4c1cdfd72938ed47730a0f87d15ce20257e" gracePeriod=2 Sep 10 14:55:56.308718 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.308687 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.314170 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.314145 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks"] Sep 10 14:55:56.319665 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.319642 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:55:56.320000 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.319987 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" containerName="manager" Sep 10 14:55:56.320050 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.320002 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" containerName="manager" Sep 10 14:55:56.320050 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.320015 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" containerName="manager" Sep 10 14:55:56.320050 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.320021 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" containerName="manager" Sep 10 14:55:56.320145 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.320078 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" containerName="manager" Sep 10 14:55:56.320145 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.320088 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" containerName="manager" Sep 10 14:55:56.322156 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.322136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.327867 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.327843 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x"] Sep 10 14:55:56.328049 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.328028 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" containerName="manager" containerID="cri-o://d95980ac46c2a17c88be802f0cdc2d7f8dd86a617e3c6e6034c1d31957b0f6e7" gracePeriod=2 Sep 10 14:55:56.332649 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.332626 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:55:56.334939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.334916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.338553 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.338525 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:55:56.344240 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.344214 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x"] Sep 10 14:55:56.357101 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.355523 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:55:56.359386 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.358412 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8"] Sep 10 14:55:56.359386 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.358889 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" containerName="manager" Sep 10 14:55:56.359386 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.358907 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" containerName="manager" Sep 10 14:55:56.359386 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.359014 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" containerName="manager" Sep 10 14:55:56.362121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.361916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:56.363441 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.363413 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.365326 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.365298 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.367353 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.367324 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:56.369398 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.369351 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.371892 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.371850 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8"] Sep 10 14:55:56.390598 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.390512 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.400722 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.400694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6ld\" (UniqueName: \"kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.400867 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.400777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p422w\" (UniqueName: \"kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.400867 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.400831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78skj\" (UniqueName: \"kubernetes.io/projected/e73d8582-ee86-4538-adca-1967a290050d-kube-api-access-78skj\") pod \"limitador-operator-controller-manager-74495c69f8-9lvc8\" (UID: \"e73d8582-ee86-4538-adca-1967a290050d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:56.400981 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.400892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.400981 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.400945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.458112 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.458021 2581 generic.go:356] "Generic (PLEG): container finished" podID="9bdea041-d0d1-4241-b57f-63f4db525bc1" containerID="583baba0f6c960be01c3767ec7c0203d4eaa513347219a8da351ef1f20a7c386" exitCode=0 Sep 10 14:55:56.460641 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.460603 2581 generic.go:356] "Generic (PLEG): container finished" podID="a878fb2d-527a-4969-b7ff-3c3ece425121" containerID="d95980ac46c2a17c88be802f0cdc2d7f8dd86a617e3c6e6034c1d31957b0f6e7" exitCode=0 Sep 10 14:55:56.463090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.463064 2581 generic.go:356] "Generic (PLEG): container finished" podID="5579ed59-05e1-4b50-84ca-d3c312c642d2" containerID="709ff1c8b29252080df2d807735fb4c1cdfd72938ed47730a0f87d15ce20257e" exitCode=0 Sep 10 14:55:56.502798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.502315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78skj\" (UniqueName: \"kubernetes.io/projected/e73d8582-ee86-4538-adca-1967a290050d-kube-api-access-78skj\") pod \"limitador-operator-controller-manager-74495c69f8-9lvc8\" (UID: \"e73d8582-ee86-4538-adca-1967a290050d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:56.502798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.502376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.502798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.502431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.502798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.502482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6ld\" (UniqueName: \"kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.502798 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.502530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p422w\" (UniqueName: \"kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.503550 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.503241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.503550 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.503505 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.513261 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.513234 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p422w\" (UniqueName: \"kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-sxk45\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.513603 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.513575 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6ld\" (UniqueName: \"kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld\") pod \"kuadrant-operator-controller-manager-76f8f686bf-46rj7\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.513922 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.513898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78skj\" (UniqueName: \"kubernetes.io/projected/e73d8582-ee86-4538-adca-1967a290050d-kube-api-access-78skj\") pod \"limitador-operator-controller-manager-74495c69f8-9lvc8\" (UID: \"e73d8582-ee86-4538-adca-1967a290050d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:56.573244 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.573214 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.575368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.575347 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.575478 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.575456 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.577047 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.577026 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.578675 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.578654 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:56.580347 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.580323 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:56.580644 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.580630 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:56.582072 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.582048 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.583566 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.583548 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.585129 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.585106 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:56.586780 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.586746 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:56.588418 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:56.588398 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:56.703451 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703418 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume\") pod \"5579ed59-05e1-4b50-84ca-d3c312c642d2\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " Sep 10 14:55:56.703634 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703474 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cdrk\" (UniqueName: \"kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk\") pod \"a878fb2d-527a-4969-b7ff-3c3ece425121\" (UID: \"a878fb2d-527a-4969-b7ff-3c3ece425121\") " Sep 10 14:55:56.703634 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703518 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume\") pod \"9bdea041-d0d1-4241-b57f-63f4db525bc1\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " Sep 10 14:55:56.703634 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls49k\" (UniqueName: \"kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k\") pod \"9bdea041-d0d1-4241-b57f-63f4db525bc1\" (UID: \"9bdea041-d0d1-4241-b57f-63f4db525bc1\") " Sep 10 14:55:56.703634 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703603 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5zvc\" (UniqueName: \"kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc\") pod \"5579ed59-05e1-4b50-84ca-d3c312c642d2\" (UID: \"5579ed59-05e1-4b50-84ca-d3c312c642d2\") " Sep 10 14:55:56.703917 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5579ed59-05e1-4b50-84ca-d3c312c642d2" (UID: "5579ed59-05e1-4b50-84ca-d3c312c642d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:55:56.703981 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.703960 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9bdea041-d0d1-4241-b57f-63f4db525bc1" (UID: "9bdea041-d0d1-4241-b57f-63f4db525bc1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:55:56.705943 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.705921 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k" (OuterVolumeSpecName: "kube-api-access-ls49k") pod "9bdea041-d0d1-4241-b57f-63f4db525bc1" (UID: "9bdea041-d0d1-4241-b57f-63f4db525bc1"). InnerVolumeSpecName "kube-api-access-ls49k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:55:56.706017 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.705966 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk" (OuterVolumeSpecName: "kube-api-access-7cdrk") pod "a878fb2d-527a-4969-b7ff-3c3ece425121" (UID: "a878fb2d-527a-4969-b7ff-3c3ece425121"). InnerVolumeSpecName "kube-api-access-7cdrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:55:56.706060 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.706014 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc" (OuterVolumeSpecName: "kube-api-access-n5zvc") pod "5579ed59-05e1-4b50-84ca-d3c312c642d2" (UID: "5579ed59-05e1-4b50-84ca-d3c312c642d2"). InnerVolumeSpecName "kube-api-access-n5zvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:55:56.762236 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.762197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:56.778474 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.778446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:56.784629 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.784601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:56.805298 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.805264 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls49k\" (UniqueName: \"kubernetes.io/projected/9bdea041-d0d1-4241-b57f-63f4db525bc1-kube-api-access-ls49k\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:55:56.805298 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.805303 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zvc\" (UniqueName: \"kubernetes.io/projected/5579ed59-05e1-4b50-84ca-d3c312c642d2-kube-api-access-n5zvc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:55:56.805486 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.805319 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5579ed59-05e1-4b50-84ca-d3c312c642d2-extensions-socket-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:55:56.805486 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.805333 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdrk\" (UniqueName: \"kubernetes.io/projected/a878fb2d-527a-4969-b7ff-3c3ece425121-kube-api-access-7cdrk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:55:56.805486 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.805347 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bdea041-d0d1-4241-b57f-63f4db525bc1-extensions-socket-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:55:56.895090 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.895052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:56.990444 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:56.990413 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:55:57.039418 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.039386 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:55:57.042513 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:57.042490 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae59f80_717d_48bb_bd14_347b54989603.slice/crio-67f1e893e6de0e9700e69cfe18e43ed7fe267b3d7dcd4ae05caffa4297305f7b WatchSource:0}: Error finding container 67f1e893e6de0e9700e69cfe18e43ed7fe267b3d7dcd4ae05caffa4297305f7b: Status 404 returned error can't find the container with id 67f1e893e6de0e9700e69cfe18e43ed7fe267b3d7dcd4ae05caffa4297305f7b Sep 10 14:55:57.077396 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.077329 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8"] Sep 10 14:55:57.080172 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:57.080040 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73d8582_ee86_4538_adca_1967a290050d.slice/crio-b14e85aa0d39d2bc0f534b4cfe6186a1fd49033b7b4901b33a6209e679af2923 WatchSource:0}: Error finding container b14e85aa0d39d2bc0f534b4cfe6186a1fd49033b7b4901b33a6209e679af2923: Status 404 returned error can't find the container with id b14e85aa0d39d2bc0f534b4cfe6186a1fd49033b7b4901b33a6209e679af2923 Sep 10 14:55:57.081377 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.081353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:55:57.083653 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:55:57.083628 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde558699_4387_4a6e_b890_c04ccfe2e5d9.slice/crio-d974027caf33046174822450f0abbec99094977d2e6f3c9710f3d96554d900dd WatchSource:0}: Error finding container d974027caf33046174822450f0abbec99094977d2e6f3c9710f3d96554d900dd: Status 404 returned error can't find the container with id d974027caf33046174822450f0abbec99094977d2e6f3c9710f3d96554d900dd Sep 10 14:55:57.470517 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.470426 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:57.470947 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.470407 2581 scope.go:117] "RemoveContainer" containerID="d95980ac46c2a17c88be802f0cdc2d7f8dd86a617e3c6e6034c1d31957b0f6e7" Sep 10 14:55:57.472461 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.472426 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" event={"ID":"fae59f80-717d-48bb-bd14-347b54989603","Type":"ContainerStarted","Data":"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b"} Sep 10 14:55:57.472590 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.472529 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:55:57.472590 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.472558 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" event={"ID":"fae59f80-717d-48bb-bd14-347b54989603","Type":"ContainerStarted","Data":"67f1e893e6de0e9700e69cfe18e43ed7fe267b3d7dcd4ae05caffa4297305f7b"} Sep 10 14:55:57.472711 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.472587 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:57.474087 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.474066 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:57.474586 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.474547 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:57.475717 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.475697 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:57.476494 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.476444 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:57.477936 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.477909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" event={"ID":"e73d8582-ee86-4538-adca-1967a290050d","Type":"ContainerStarted","Data":"789acf5411bbb0f65a6661c84fd2041f5464b90e6c8ba742a9da8a14d9fb14a0"} Sep 10 14:55:57.478023 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.477960 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" event={"ID":"e73d8582-ee86-4538-adca-1967a290050d","Type":"ContainerStarted","Data":"b14e85aa0d39d2bc0f534b4cfe6186a1fd49033b7b4901b33a6209e679af2923"} Sep 10 14:55:57.478023 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.477988 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:55:57.478444 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.478421 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:57.480398 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.480284 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" event={"ID":"de558699-4387-4a6e-b890-c04ccfe2e5d9","Type":"ContainerStarted","Data":"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd"} Sep 10 14:55:57.480398 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.480334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" event={"ID":"de558699-4387-4a6e-b890-c04ccfe2e5d9","Type":"ContainerStarted","Data":"d974027caf33046174822450f0abbec99094977d2e6f3c9710f3d96554d900dd"} Sep 10 14:55:57.480398 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.480364 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:55:57.481385 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.481371 2581 scope.go:117] "RemoveContainer" containerID="709ff1c8b29252080df2d807735fb4c1cdfd72938ed47730a0f87d15ce20257e" Sep 10 14:55:57.491267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.491245 2581 scope.go:117] "RemoveContainer" containerID="583baba0f6c960be01c3767ec7c0203d4eaa513347219a8da351ef1f20a7c386" Sep 10 14:55:57.499857 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.499810 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" podStartSLOduration=1.4997930959999999 podStartE2EDuration="1.499793096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:57.498113408 +0000 UTC m=+511.792415539" watchObservedRunningTime="2026-09-10 14:55:57.499793096 +0000 UTC m=+511.794095221" Sep 10 14:55:57.500074 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.500048 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:57.501706 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.501680 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:57.503398 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.503375 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-v7vt6\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-v7vt6" Sep 10 14:55:57.523674 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.523610 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" podStartSLOduration=1.523591855 podStartE2EDuration="1.523591855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:57.522973929 +0000 UTC m=+511.817276060" watchObservedRunningTime="2026-09-10 14:55:57.523591855 +0000 UTC m=+511.817893984" Sep 10 14:55:57.524945 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.524921 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-7wrks\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wrks" Sep 10 14:55:57.541835 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:57.541777 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" podStartSLOduration=1.5417436690000001 podStartE2EDuration="1.541743669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:57.540480664 +0000 UTC m=+511.834782792" watchObservedRunningTime="2026-09-10 14:55:57.541743669 +0000 UTC m=+511.836045797" Sep 10 14:55:57.542531 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:55:57.542506 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-r4l2x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4l2x" Sep 10 14:55:58.230368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:58.230334 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5579ed59-05e1-4b50-84ca-d3c312c642d2" path="/var/lib/kubelet/pods/5579ed59-05e1-4b50-84ca-d3c312c642d2/volumes" Sep 10 14:55:58.230685 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:58.230672 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdea041-d0d1-4241-b57f-63f4db525bc1" path="/var/lib/kubelet/pods/9bdea041-d0d1-4241-b57f-63f4db525bc1/volumes" Sep 10 14:55:58.230977 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:55:58.230965 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a878fb2d-527a-4969-b7ff-3c3ece425121" path="/var/lib/kubelet/pods/a878fb2d-527a-4969-b7ff-3c3ece425121/volumes" Sep 10 14:56:01.615387 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:01.615353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:01.716942 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:01.716889 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:06.420282 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:06.420248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:06.520095 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:06.520064 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:07.536854 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:07.536819 2581 generic.go:356] "Generic (PLEG): container finished" podID="e73d8582-ee86-4538-adca-1967a290050d" containerID="789acf5411bbb0f65a6661c84fd2041f5464b90e6c8ba742a9da8a14d9fb14a0" exitCode=1 Sep 10 14:56:07.537224 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:07.536895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" event={"ID":"e73d8582-ee86-4538-adca-1967a290050d","Type":"ContainerDied","Data":"789acf5411bbb0f65a6661c84fd2041f5464b90e6c8ba742a9da8a14d9fb14a0"} Sep 10 14:56:07.537283 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:07.537267 2581 scope.go:117] "RemoveContainer" containerID="789acf5411bbb0f65a6661c84fd2041f5464b90e6c8ba742a9da8a14d9fb14a0" Sep 10 14:56:08.491547 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.491515 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:56:08.491728 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.491587 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:56:08.545508 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.545474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" event={"ID":"e73d8582-ee86-4538-adca-1967a290050d","Type":"ContainerStarted","Data":"588dbb943e0f5f33da086ef564787b8148d351e6453d3d9a9cc7661fb8e5b9c2"} Sep 10 14:56:08.545906 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.545685 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:56:08.557506 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.557479 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:56:08.557688 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.557646 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" podUID="de558699-4387-4a6e-b890-c04ccfe2e5d9" containerName="manager" containerID="cri-o://83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd" gracePeriod=10 Sep 10 14:56:08.799493 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.799468 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:56:08.909825 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.909781 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 14:56:08.910177 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.910159 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de558699-4387-4a6e-b890-c04ccfe2e5d9" containerName="manager" Sep 10 14:56:08.910267 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.910182 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de558699-4387-4a6e-b890-c04ccfe2e5d9" containerName="manager" Sep 10 14:56:08.910318 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.910289 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de558699-4387-4a6e-b890-c04ccfe2e5d9" containerName="manager" Sep 10 14:56:08.911294 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.911270 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume\") pod \"de558699-4387-4a6e-b890-c04ccfe2e5d9\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " Sep 10 14:56:08.911424 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.911311 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz6ld\" (UniqueName: \"kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld\") pod \"de558699-4387-4a6e-b890-c04ccfe2e5d9\" (UID: \"de558699-4387-4a6e-b890-c04ccfe2e5d9\") " Sep 10 14:56:08.911660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.911637 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "de558699-4387-4a6e-b890-c04ccfe2e5d9" (UID: "de558699-4387-4a6e-b890-c04ccfe2e5d9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:56:08.912325 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.912304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:08.913891 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.913868 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld" (OuterVolumeSpecName: "kube-api-access-gz6ld") pod "de558699-4387-4a6e-b890-c04ccfe2e5d9" (UID: "de558699-4387-4a6e-b890-c04ccfe2e5d9"). InnerVolumeSpecName "kube-api-access-gz6ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:56:08.925350 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:08.925328 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 14:56:09.011872 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.011778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkh8\" (UniqueName: \"kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.011872 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.011831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.012065 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.011948 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de558699-4387-4a6e-b890-c04ccfe2e5d9-extensions-socket-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:09.012065 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.011963 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6ld\" (UniqueName: \"kubernetes.io/projected/de558699-4387-4a6e-b890-c04ccfe2e5d9-kube-api-access-gz6ld\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:09.112958 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.112910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkh8\" (UniqueName: \"kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.112958 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.112967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.113301 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.113283 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.121380 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.121350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkh8\" (UniqueName: \"kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-g779h\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.228405 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.228374 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:09.503455 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.503405 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 14:56:09.505708 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:56:09.505683 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cea6fd3_e290_460b_9010_cab131fdd0bb.slice/crio-c59cf0f2b6dea014d8c4e6fc229dc979c3ea3598ef590666a127909326ce572e WatchSource:0}: Error finding container c59cf0f2b6dea014d8c4e6fc229dc979c3ea3598ef590666a127909326ce572e: Status 404 returned error can't find the container with id c59cf0f2b6dea014d8c4e6fc229dc979c3ea3598ef590666a127909326ce572e Sep 10 14:56:09.552336 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.552312 2581 generic.go:356] "Generic (PLEG): container finished" podID="de558699-4387-4a6e-b890-c04ccfe2e5d9" containerID="83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd" exitCode=0 Sep 10 14:56:09.552701 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.552356 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" Sep 10 14:56:09.552701 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.552394 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" event={"ID":"de558699-4387-4a6e-b890-c04ccfe2e5d9","Type":"ContainerDied","Data":"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd"} Sep 10 14:56:09.552701 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.552443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7" event={"ID":"de558699-4387-4a6e-b890-c04ccfe2e5d9","Type":"ContainerDied","Data":"d974027caf33046174822450f0abbec99094977d2e6f3c9710f3d96554d900dd"} Sep 10 14:56:09.552701 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.552467 2581 scope.go:117] "RemoveContainer" containerID="83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd" Sep 10 14:56:09.554187 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.554167 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" event={"ID":"6cea6fd3-e290-460b-9010-cab131fdd0bb","Type":"ContainerStarted","Data":"c59cf0f2b6dea014d8c4e6fc229dc979c3ea3598ef590666a127909326ce572e"} Sep 10 14:56:09.561148 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.561128 2581 scope.go:117] "RemoveContainer" containerID="83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd" Sep 10 14:56:09.561403 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:56:09.561384 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd\": container with ID starting with 83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd not found: ID does not exist" containerID="83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd" Sep 10 14:56:09.561452 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.561411 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd"} err="failed to get container status \"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd\": rpc error: code = NotFound desc = could not find container \"83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd\": container with ID starting with 83f558028821e9c147f74b097d30ef40e923c414dfe20cd6007a8540e16874bd not found: ID does not exist" Sep 10 14:56:09.576119 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.576094 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:56:09.578164 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:09.578146 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-46rj7"] Sep 10 14:56:10.229315 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:10.229279 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de558699-4387-4a6e-b890-c04ccfe2e5d9" path="/var/lib/kubelet/pods/de558699-4387-4a6e-b890-c04ccfe2e5d9/volumes" Sep 10 14:56:10.561400 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:10.561365 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" event={"ID":"6cea6fd3-e290-460b-9010-cab131fdd0bb","Type":"ContainerStarted","Data":"e4aaf3ad9bb50c6c0ad3ce4adfb4b746cb1006d663c8ff9443c0c037027d5d44"} Sep 10 14:56:10.561818 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:10.561507 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:10.585419 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:10.585361 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" podStartSLOduration=2.585343801 podStartE2EDuration="2.585343801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:10.584250399 +0000 UTC m=+524.878552528" watchObservedRunningTime="2026-09-10 14:56:10.585343801 +0000 UTC m=+524.879645930" Sep 10 14:56:19.557145 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:19.557072 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lvc8" Sep 10 14:56:21.568241 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.568208 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 14:56:21.615477 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.615444 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:56:21.615709 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.615679 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" podUID="fae59f80-717d-48bb-bd14-347b54989603" containerName="manager" containerID="cri-o://00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b" gracePeriod=10 Sep 10 14:56:21.864399 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.864375 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:56:21.924501 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.924460 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p422w\" (UniqueName: \"kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w\") pod \"fae59f80-717d-48bb-bd14-347b54989603\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " Sep 10 14:56:21.924669 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.924518 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume\") pod \"fae59f80-717d-48bb-bd14-347b54989603\" (UID: \"fae59f80-717d-48bb-bd14-347b54989603\") " Sep 10 14:56:21.924928 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.924907 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fae59f80-717d-48bb-bd14-347b54989603" (UID: "fae59f80-717d-48bb-bd14-347b54989603"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:56:21.926831 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:21.926807 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w" (OuterVolumeSpecName: "kube-api-access-p422w") pod "fae59f80-717d-48bb-bd14-347b54989603" (UID: "fae59f80-717d-48bb-bd14-347b54989603"). InnerVolumeSpecName "kube-api-access-p422w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:56:22.026063 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.026021 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p422w\" (UniqueName: \"kubernetes.io/projected/fae59f80-717d-48bb-bd14-347b54989603-kube-api-access-p422w\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:22.026063 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.026054 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fae59f80-717d-48bb-bd14-347b54989603-extensions-socket-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:22.627107 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.627008 2581 generic.go:356] "Generic (PLEG): container finished" podID="fae59f80-717d-48bb-bd14-347b54989603" containerID="00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b" exitCode=0 Sep 10 14:56:22.627107 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.627066 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" Sep 10 14:56:22.627107 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.627084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" event={"ID":"fae59f80-717d-48bb-bd14-347b54989603","Type":"ContainerDied","Data":"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b"} Sep 10 14:56:22.627575 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.627158 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45" event={"ID":"fae59f80-717d-48bb-bd14-347b54989603","Type":"ContainerDied","Data":"67f1e893e6de0e9700e69cfe18e43ed7fe267b3d7dcd4ae05caffa4297305f7b"} Sep 10 14:56:22.627575 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.627181 2581 scope.go:117] "RemoveContainer" containerID="00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b" Sep 10 14:56:22.636227 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.636207 2581 scope.go:117] "RemoveContainer" containerID="00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b" Sep 10 14:56:22.636561 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:56:22.636541 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b\": container with ID starting with 00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b not found: ID does not exist" containerID="00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b" Sep 10 14:56:22.636615 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.636569 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b"} err="failed to get container status \"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b\": rpc error: code = NotFound desc = could not find container \"00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b\": container with ID starting with 00e3b9cabbf4026bdfbbfc91f6b6b6e943fff80fcecc6eb5cbf9b19ffd0c7d6b not found: ID does not exist" Sep 10 14:56:22.645052 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.645018 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:56:22.649906 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:22.649877 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-sxk45"] Sep 10 14:56:24.229179 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:24.229150 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae59f80-717d-48bb-bd14-347b54989603" path="/var/lib/kubelet/pods/fae59f80-717d-48bb-bd14-347b54989603/volumes" Sep 10 14:56:24.878533 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:24.878503 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:25.573245 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:25.573209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:38.940013 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.939973 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:38.940625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.940382 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae59f80-717d-48bb-bd14-347b54989603" containerName="manager" Sep 10 14:56:38.940625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.940403 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae59f80-717d-48bb-bd14-347b54989603" containerName="manager" Sep 10 14:56:38.940625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.940514 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae59f80-717d-48bb-bd14-347b54989603" containerName="manager" Sep 10 14:56:38.943408 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.943376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:38.945226 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.945207 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:38.947580 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.947561 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 14:56:38.950417 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:38.950393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:39.042904 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.042862 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:39.053501 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.053466 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:39.061461 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.061430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.061660 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.061488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsnv\" (UniqueName: \"kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.113989 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.113953 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:39.162959 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.162920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.163134 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.162973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsnv\" (UniqueName: \"kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.163622 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.163601 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.171052 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.171022 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsnv\" (UniqueName: \"kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv\") pod \"limitador-limitador-8588749977-7tc9k\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.256591 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.256547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:39.529031 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.528847 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:39.531770 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:56:39.531712 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e4d499_2d1e_4ab0_9c92_e10cf415542e.slice/crio-af060eb22a7918974d3aec0dfd0accbaa2e9ea20033fb0606d9c7401a8fbc95f WatchSource:0}: Error finding container af060eb22a7918974d3aec0dfd0accbaa2e9ea20033fb0606d9c7401a8fbc95f: Status 404 returned error can't find the container with id af060eb22a7918974d3aec0dfd0accbaa2e9ea20033fb0606d9c7401a8fbc95f Sep 10 14:56:39.706768 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.706732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" event={"ID":"61e4d499-2d1e-4ab0-9c92-e10cf415542e","Type":"ContainerStarted","Data":"af060eb22a7918974d3aec0dfd0accbaa2e9ea20033fb0606d9c7401a8fbc95f"} Sep 10 14:56:39.728943 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.728907 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:39.733461 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.733443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:39.735610 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.735589 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qznsd\"" Sep 10 14:56:39.738484 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.738459 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:39.759593 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.759544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:39.768678 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.768646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm9z\" (UniqueName: \"kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z\") pod \"authorino-7bcfd8f9cd-j89sn\" (UID: \"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:39.849481 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.849401 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:39.869925 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.869890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm9z\" (UniqueName: \"kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z\") pod \"authorino-7bcfd8f9cd-j89sn\" (UID: \"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:39.877835 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:39.877802 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm9z\" (UniqueName: \"kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z\") pod \"authorino-7bcfd8f9cd-j89sn\" (UID: \"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:40.045004 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.044965 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:40.313265 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.313198 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:40.465466 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:56:40.465424 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd08ec17_80c0_4eb4_bf0c_c970f5c553c0.slice/crio-93efc901c162da27b1d76514e8536305182e019db56e9ab0fbb9b010a21557f1 WatchSource:0}: Error finding container 93efc901c162da27b1d76514e8536305182e019db56e9ab0fbb9b010a21557f1: Status 404 returned error can't find the container with id 93efc901c162da27b1d76514e8536305182e019db56e9ab0fbb9b010a21557f1 Sep 10 14:56:40.712666 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.712627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" event={"ID":"61e4d499-2d1e-4ab0-9c92-e10cf415542e","Type":"ContainerStarted","Data":"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098"} Sep 10 14:56:40.712881 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.712685 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:40.713809 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.713782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" event={"ID":"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0","Type":"ContainerStarted","Data":"93efc901c162da27b1d76514e8536305182e019db56e9ab0fbb9b010a21557f1"} Sep 10 14:56:40.728508 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:40.728458 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" podStartSLOduration=1.7409565489999999 podStartE2EDuration="2.728442018s" totalImagesPullingTime="987.485469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:40.727508259 +0000 UTC m=+555.021810388" watchObservedRunningTime="2026-09-10 14:56:40.728442018 +0000 UTC m=+555.022744148" Sep 10 14:56:42.726127 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:42.726032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" event={"ID":"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0","Type":"ContainerStarted","Data":"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e"} Sep 10 14:56:42.740404 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:42.740352 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" podStartSLOduration=1.901942899 podStartE2EDuration="3.740334457s" totalImagesPullingTime="1.838391558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:42.739500243 +0000 UTC m=+557.033802371" watchObservedRunningTime="2026-09-10 14:56:42.740334457 +0000 UTC m=+557.034636586" Sep 10 14:56:43.466728 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:43.466692 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:44.734832 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:44.734793 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" podUID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" containerName="authorino" containerID="cri-o://d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e" gracePeriod=30 Sep 10 14:56:44.982288 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:44.982258 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:45.116984 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.116948 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzm9z\" (UniqueName: \"kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z\") pod \"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0\" (UID: \"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0\") " Sep 10 14:56:45.119169 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.119133 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z" (OuterVolumeSpecName: "kube-api-access-fzm9z") pod "fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" (UID: "fd08ec17-80c0-4eb4-bf0c-c970f5c553c0"). InnerVolumeSpecName "kube-api-access-fzm9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:56:45.218362 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.218310 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzm9z\" (UniqueName: \"kubernetes.io/projected/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0-kube-api-access-fzm9z\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:45.741342 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.741312 2581 generic.go:356] "Generic (PLEG): container finished" podID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" containerID="d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e" exitCode=0 Sep 10 14:56:45.741788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.741352 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" Sep 10 14:56:45.741788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.741373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" event={"ID":"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0","Type":"ContainerDied","Data":"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e"} Sep 10 14:56:45.741788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.741407 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j89sn" event={"ID":"fd08ec17-80c0-4eb4-bf0c-c970f5c553c0","Type":"ContainerDied","Data":"93efc901c162da27b1d76514e8536305182e019db56e9ab0fbb9b010a21557f1"} Sep 10 14:56:45.741788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.741425 2581 scope.go:117] "RemoveContainer" containerID="d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e" Sep 10 14:56:45.751967 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.751942 2581 scope.go:117] "RemoveContainer" containerID="d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e" Sep 10 14:56:45.752491 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:56:45.752470 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e\": container with ID starting with d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e not found: ID does not exist" containerID="d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e" Sep 10 14:56:45.752577 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.752505 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e"} err="failed to get container status \"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e\": rpc error: code = NotFound desc = could not find container \"d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e\": container with ID starting with d91d59bcb523889e41331ab9637d2611ad6059ecabff80e751ce759aab02316e not found: ID does not exist" Sep 10 14:56:45.764169 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.764135 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:45.766202 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:45.766175 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j89sn"] Sep 10 14:56:46.229849 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:46.229774 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" path="/var/lib/kubelet/pods/fd08ec17-80c0-4eb4-bf0c-c970f5c553c0/volumes" Sep 10 14:56:51.721331 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:51.721297 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:54.479796 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:54.479745 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:54.480196 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:54.479947 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" podUID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" containerName="limitador" containerID="cri-o://5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098" gracePeriod=30 Sep 10 14:56:55.028695 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.028668 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:55.101379 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.101285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file\") pod \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " Sep 10 14:56:55.101553 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.101485 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlsnv\" (UniqueName: \"kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv\") pod \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\" (UID: \"61e4d499-2d1e-4ab0-9c92-e10cf415542e\") " Sep 10 14:56:55.101710 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.101675 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file" (OuterVolumeSpecName: "config-file") pod "61e4d499-2d1e-4ab0-9c92-e10cf415542e" (UID: "61e4d499-2d1e-4ab0-9c92-e10cf415542e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:56:55.104258 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.104221 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv" (OuterVolumeSpecName: "kube-api-access-hlsnv") pod "61e4d499-2d1e-4ab0-9c92-e10cf415542e" (UID: "61e4d499-2d1e-4ab0-9c92-e10cf415542e"). InnerVolumeSpecName "kube-api-access-hlsnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:56:55.202968 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.202934 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61e4d499-2d1e-4ab0-9c92-e10cf415542e-config-file\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:55.202968 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.202964 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsnv\" (UniqueName: \"kubernetes.io/projected/61e4d499-2d1e-4ab0-9c92-e10cf415542e-kube-api-access-hlsnv\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:56:55.789151 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.789117 2581 generic.go:356] "Generic (PLEG): container finished" podID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" containerID="5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098" exitCode=0 Sep 10 14:56:55.789610 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.789174 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" Sep 10 14:56:55.789610 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.789185 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" event={"ID":"61e4d499-2d1e-4ab0-9c92-e10cf415542e","Type":"ContainerDied","Data":"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098"} Sep 10 14:56:55.789610 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.789229 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7tc9k" event={"ID":"61e4d499-2d1e-4ab0-9c92-e10cf415542e","Type":"ContainerDied","Data":"af060eb22a7918974d3aec0dfd0accbaa2e9ea20033fb0606d9c7401a8fbc95f"} Sep 10 14:56:55.789610 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.789251 2581 scope.go:117] "RemoveContainer" containerID="5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098" Sep 10 14:56:55.798546 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.798529 2581 scope.go:117] "RemoveContainer" containerID="5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098" Sep 10 14:56:55.798858 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:56:55.798836 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098\": container with ID starting with 5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098 not found: ID does not exist" containerID="5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098" Sep 10 14:56:55.798918 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.798867 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098"} err="failed to get container status \"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098\": rpc error: code = NotFound desc = could not find container \"5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098\": container with ID starting with 5c117c41bdfcb4c021329677650b884f4564e0d7b1ee799581b5f5648408f098 not found: ID does not exist" Sep 10 14:56:55.811413 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.811387 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:55.816788 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:55.816751 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7tc9k"] Sep 10 14:56:56.230024 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:56.229945 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" path="/var/lib/kubelet/pods/61e4d499-2d1e-4ab0-9c92-e10cf415542e/volumes" Sep 10 14:56:58.420221 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.420190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:58.853981 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.853951 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nxznl"] Sep 10 14:56:58.854299 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854287 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" containerName="limitador" Sep 10 14:56:58.854345 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854302 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" containerName="limitador" Sep 10 14:56:58.854345 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854320 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" containerName="authorino" Sep 10 14:56:58.854345 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854326 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" containerName="authorino" Sep 10 14:56:58.854438 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854376 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e4d499-2d1e-4ab0-9c92-e10cf415542e" containerName="limitador" Sep 10 14:56:58.854438 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.854393 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd08ec17-80c0-4eb4-bf0c-c970f5c553c0" containerName="authorino" Sep 10 14:56:58.859322 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.859301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:58.861979 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.861948 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-654l6\"" Sep 10 14:56:58.862126 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.862107 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:56:58.862847 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.862827 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 14:56:58.862948 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.862909 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 14:56:58.867460 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.867436 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nxznl"] Sep 10 14:56:58.934161 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.934119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af61b48a-d7b0-4b33-bdce-f6b285c8c819-data\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:58.934347 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:58.934177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44d9s\" (UniqueName: \"kubernetes.io/projected/af61b48a-d7b0-4b33-bdce-f6b285c8c819-kube-api-access-44d9s\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.035494 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.035452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af61b48a-d7b0-4b33-bdce-f6b285c8c819-data\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.035706 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.035507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44d9s\" (UniqueName: \"kubernetes.io/projected/af61b48a-d7b0-4b33-bdce-f6b285c8c819-kube-api-access-44d9s\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.035961 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.035937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af61b48a-d7b0-4b33-bdce-f6b285c8c819-data\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.043618 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.043595 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44d9s\" (UniqueName: \"kubernetes.io/projected/af61b48a-d7b0-4b33-bdce-f6b285c8c819-kube-api-access-44d9s\") pod \"postgres-69fb4594cf-nxznl\" (UID: \"af61b48a-d7b0-4b33-bdce-f6b285c8c819\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.171680 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.171603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:56:59.445185 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.445110 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nxznl"] Sep 10 14:56:59.448882 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:56:59.448854 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf61b48a_d7b0_4b33_bdce_f6b285c8c819.slice/crio-6c245061a065a4c5e2aace6806cb46ce01b7ec2623251b5158373d03cd38f761 WatchSource:0}: Error finding container 6c245061a065a4c5e2aace6806cb46ce01b7ec2623251b5158373d03cd38f761: Status 404 returned error can't find the container with id 6c245061a065a4c5e2aace6806cb46ce01b7ec2623251b5158373d03cd38f761 Sep 10 14:56:59.809621 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.809590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" event={"ID":"af61b48a-d7b0-4b33-bdce-f6b285c8c819","Type":"ContainerStarted","Data":"6c245061a065a4c5e2aace6806cb46ce01b7ec2623251b5158373d03cd38f761"} Sep 10 14:56:59.878380 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:56:59.878346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:01.821240 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:01.821150 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" event={"ID":"af61b48a-d7b0-4b33-bdce-f6b285c8c819","Type":"ContainerStarted","Data":"1700154c29c00f437a2fceca70c2772599307c2e469965bc2180e79abf6e786d"} Sep 10 14:57:01.821240 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:01.821221 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:57:01.836810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:01.836741 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" podStartSLOduration=1.755536298 podStartE2EDuration="3.836722368s" totalImagesPullingTime="2.08118607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:01.835398287 +0000 UTC m=+576.129700417" watchObservedRunningTime="2026-09-10 14:57:01.836722368 +0000 UTC m=+576.131024496" Sep 10 14:57:07.859662 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:07.859630 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nxznl" Sep 10 14:57:08.158886 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.158798 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:08.390900 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.390866 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-skqdp"] Sep 10 14:57:08.394528 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.394475 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.396612 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.396591 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qznsd\"" Sep 10 14:57:08.402856 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.402831 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-skqdp"] Sep 10 14:57:08.518933 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.518891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddlf\" (UniqueName: \"kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf\") pod \"authorino-557f876f69-skqdp\" (UID: \"c941a5de-6915-4dc7-9e94-575dedd39d93\") " pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.540137 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.540100 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:08.619825 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.619778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddlf\" (UniqueName: \"kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf\") pod \"authorino-557f876f69-skqdp\" (UID: \"c941a5de-6915-4dc7-9e94-575dedd39d93\") " pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.622968 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.622935 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-skqdp"] Sep 10 14:57:08.623193 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:08.623174 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lddlf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-skqdp" podUID="c941a5de-6915-4dc7-9e94-575dedd39d93" Sep 10 14:57:08.631971 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.631936 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddlf\" (UniqueName: \"kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf\") pod \"authorino-557f876f69-skqdp\" (UID: \"c941a5de-6915-4dc7-9e94-575dedd39d93\") " pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.650258 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.650226 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 14:57:08.653935 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.653908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.656140 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.656119 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 14:57:08.660362 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.660334 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 14:57:08.822310 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.822208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8x4\" (UniqueName: \"kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.822310 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.822261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.862577 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.862540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.869451 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.869428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:08.923383 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.923344 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8x4\" (UniqueName: \"kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.923578 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.923395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.926339 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.926315 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.931836 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.931811 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8x4\" (UniqueName: \"kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4\") pod \"authorino-94769c5bf-skr8s\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:08.964067 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:08.964030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 14:57:09.024544 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.024249 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lddlf\" (UniqueName: \"kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf\") pod \"c941a5de-6915-4dc7-9e94-575dedd39d93\" (UID: \"c941a5de-6915-4dc7-9e94-575dedd39d93\") " Sep 10 14:57:09.027408 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.027373 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf" (OuterVolumeSpecName: "kube-api-access-lddlf") pod "c941a5de-6915-4dc7-9e94-575dedd39d93" (UID: "c941a5de-6915-4dc7-9e94-575dedd39d93"). InnerVolumeSpecName "kube-api-access-lddlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:57:09.125795 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.125684 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddlf\" (UniqueName: \"kubernetes.io/projected/c941a5de-6915-4dc7-9e94-575dedd39d93-kube-api-access-lddlf\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:57:09.232397 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.232366 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 14:57:09.235937 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:57:09.235912 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef9a33bf_d252_4367_8ba4_5c446bc7ec3d.slice/crio-168e9f64a431ca99a57c4dde9c937026b6a038c42094d5551e98ec33be89e0b4 WatchSource:0}: Error finding container 168e9f64a431ca99a57c4dde9c937026b6a038c42094d5551e98ec33be89e0b4: Status 404 returned error can't find the container with id 168e9f64a431ca99a57c4dde9c937026b6a038c42094d5551e98ec33be89e0b4 Sep 10 14:57:09.869443 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.869400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94769c5bf-skr8s" event={"ID":"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d","Type":"ContainerStarted","Data":"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68"} Sep 10 14:57:09.869443 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.869449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94769c5bf-skr8s" event={"ID":"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d","Type":"ContainerStarted","Data":"168e9f64a431ca99a57c4dde9c937026b6a038c42094d5551e98ec33be89e0b4"} Sep 10 14:57:09.870046 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.869521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-skqdp" Sep 10 14:57:09.884506 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.884457 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-94769c5bf-skr8s" podStartSLOduration=1.388444403 podStartE2EDuration="1.884440427s" totalImagesPullingTime="495.996024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:09.882839262 +0000 UTC m=+584.177141391" watchObservedRunningTime="2026-09-10 14:57:09.884440427 +0000 UTC m=+584.178742556" Sep 10 14:57:09.907786 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.907686 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-skqdp"] Sep 10 14:57:09.918326 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:09.918296 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-skqdp"] Sep 10 14:57:10.229740 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:10.229665 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c941a5de-6915-4dc7-9e94-575dedd39d93" path="/var/lib/kubelet/pods/c941a5de-6915-4dc7-9e94-575dedd39d93/volumes" Sep 10 14:57:14.571515 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.571466 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:14.603199 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.603168 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:14.606603 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.606586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.609106 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.609074 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 14:57:14.609222 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.609076 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 14:57:14.609222 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.609087 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nmwpz\"" Sep 10 14:57:14.609950 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.609925 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 14:57:14.617241 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.617216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:14.646450 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.646416 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:14.703587 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.703547 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:14.772561 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.772527 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 14:57:14.776016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.775992 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.778814 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.778795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.778876 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.778837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkcp\" (UniqueName: \"kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.778876 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.778856 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.785734 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.785712 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 14:57:14.879670 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879570 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.879670 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkcp\" (UniqueName: \"kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.879834 2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.879871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.879844 2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.879909 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs podName:7c62e85b-2720-4b6d-9be4-98a2fdfb1031 nodeName:}" failed. No retries permitted until 2026-09-10 14:57:15.379884272 +0000 UTC m=+589.674186400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs") pod "maas-controller-7b9c5df8bf-9qcpv" (UID: "7c62e85b-2720-4b6d-9be4-98a2fdfb1031") : secret "maas-controller-metrics-tls" not found Sep 10 14:57:14.879955 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.879954 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert podName:7c62e85b-2720-4b6d-9be4-98a2fdfb1031 nodeName:}" failed. No retries permitted until 2026-09-10 14:57:15.379935427 +0000 UTC m=+589.674237541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert") pod "maas-controller-7b9c5df8bf-9qcpv" (UID: "7c62e85b-2720-4b6d-9be4-98a2fdfb1031") : secret "maas-controller-webhook-cert" not found Sep 10 14:57:14.890944 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.890914 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkcp\" (UniqueName: \"kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:14.980686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.980635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.980686 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.980694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.980953 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.980805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:14.980953 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.980821 2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 10 14:57:14.980953 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.980872 2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 14:57:14.980953 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.980895 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs podName:d15b707f-ce62-43ab-8d86-d879f33b9422 nodeName:}" failed. No retries permitted until 2026-09-10 14:57:15.480878382 +0000 UTC m=+589.775180489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs") pod "maas-controller-697fdc88d8-7lmtq" (UID: "d15b707f-ce62-43ab-8d86-d879f33b9422") : secret "maas-controller-metrics-tls" not found Sep 10 14:57:14.980953 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:14.980923 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert podName:d15b707f-ce62-43ab-8d86-d879f33b9422 nodeName:}" failed. No retries permitted until 2026-09-10 14:57:15.480907744 +0000 UTC m=+589.775209850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert") pod "maas-controller-697fdc88d8-7lmtq" (UID: "d15b707f-ce62-43ab-8d86-d879f33b9422") : secret "maas-controller-webhook-cert" not found Sep 10 14:57:14.989005 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:14.988973 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.088207 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.088160 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:15.314673 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.314625 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:15.384711 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.384669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:15.384940 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.384808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:15.387412 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.387379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:15.387412 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.387411 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") pod \"maas-controller-7b9c5df8bf-9qcpv\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:15.485791 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.485740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.485968 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.485819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.488394 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.488369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.488498 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.488402 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") pod \"maas-controller-697fdc88d8-7lmtq\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.518614 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.518577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:15.686890 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.686855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:15.789001 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.788977 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:15.790924 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:57:15.790896 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c62e85b_2720_4b6d_9be4_98a2fdfb1031.slice/crio-d8612e56237f050bf307481c78fdd2790f393ed4afcec95cb0787661db56bde7 WatchSource:0}: Error finding container d8612e56237f050bf307481c78fdd2790f393ed4afcec95cb0787661db56bde7: Status 404 returned error can't find the container with id d8612e56237f050bf307481c78fdd2790f393ed4afcec95cb0787661db56bde7 Sep 10 14:57:15.897613 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.897526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" event={"ID":"7c62e85b-2720-4b6d-9be4-98a2fdfb1031","Type":"ContainerStarted","Data":"d8612e56237f050bf307481c78fdd2790f393ed4afcec95cb0787661db56bde7"} Sep 10 14:57:15.958829 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:15.958804 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 14:57:15.961342 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:57:15.961312 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15b707f_ce62_43ab_8d86_d879f33b9422.slice/crio-87cb4469ae19d1b2619efc73b2c50139bc82d5ddf39cbc940f7a92a8bceb6f07 WatchSource:0}: Error finding container 87cb4469ae19d1b2619efc73b2c50139bc82d5ddf39cbc940f7a92a8bceb6f07: Status 404 returned error can't find the container with id 87cb4469ae19d1b2619efc73b2c50139bc82d5ddf39cbc940f7a92a8bceb6f07 Sep 10 14:57:16.907683 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:16.907643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" event={"ID":"d15b707f-ce62-43ab-8d86-d879f33b9422","Type":"ContainerStarted","Data":"87cb4469ae19d1b2619efc73b2c50139bc82d5ddf39cbc940f7a92a8bceb6f07"} Sep 10 14:57:18.597356 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.597303 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:18.700810 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.700687 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:18.918582 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.918546 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" event={"ID":"7c62e85b-2720-4b6d-9be4-98a2fdfb1031","Type":"ContainerStarted","Data":"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4"} Sep 10 14:57:18.918805 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.918665 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:18.920163 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.920140 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" event={"ID":"d15b707f-ce62-43ab-8d86-d879f33b9422","Type":"ContainerStarted","Data":"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff"} Sep 10 14:57:18.920270 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.920263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:18.936589 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.936542 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" podStartSLOduration=2.322013254 podStartE2EDuration="4.936525264s" totalImagesPullingTime="2.61451201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:18.935053133 +0000 UTC m=+593.229355263" watchObservedRunningTime="2026-09-10 14:57:18.936525264 +0000 UTC m=+593.230827393" Sep 10 14:57:18.952041 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:18.951947 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" podStartSLOduration=2.498114795 podStartE2EDuration="4.951924363s" totalImagesPullingTime="2.453809568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:18.951463075 +0000 UTC m=+593.245765204" watchObservedRunningTime="2026-09-10 14:57:18.951924363 +0000 UTC m=+593.246226545" Sep 10 14:57:26.158062 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:26.158027 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:57:26.158062 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:26.158051 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 14:57:29.927857 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:29.927820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:29.930372 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:29.927902 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 14:57:29.988503 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:29.988471 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:29.988706 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:29.988681 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" podUID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" containerName="manager" containerID="cri-o://617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4" gracePeriod=30 Sep 10 14:57:30.240016 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.239992 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:30.317412 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.317379 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") pod \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " Sep 10 14:57:30.317412 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.317419 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") pod \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " Sep 10 14:57:30.317614 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.317468 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mkcp\" (UniqueName: \"kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp\") pod \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\" (UID: \"7c62e85b-2720-4b6d-9be4-98a2fdfb1031\") " Sep 10 14:57:30.319804 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.319751 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7c62e85b-2720-4b6d-9be4-98a2fdfb1031" (UID: "7c62e85b-2720-4b6d-9be4-98a2fdfb1031"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:57:30.320193 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.320164 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c62e85b-2720-4b6d-9be4-98a2fdfb1031" (UID: "7c62e85b-2720-4b6d-9be4-98a2fdfb1031"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:57:30.320193 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.320178 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp" (OuterVolumeSpecName: "kube-api-access-7mkcp") pod "7c62e85b-2720-4b6d-9be4-98a2fdfb1031" (UID: "7c62e85b-2720-4b6d-9be4-98a2fdfb1031"). InnerVolumeSpecName "kube-api-access-7mkcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:57:30.418340 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.418300 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-webhook-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:57:30.418340 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.418331 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-metrics-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:57:30.418340 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.418340 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkcp\" (UniqueName: \"kubernetes.io/projected/7c62e85b-2720-4b6d-9be4-98a2fdfb1031-kube-api-access-7mkcp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 14:57:30.978946 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.978906 2581 generic.go:356] "Generic (PLEG): container finished" podID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" containerID="617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4" exitCode=0 Sep 10 14:57:30.979369 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.978959 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" Sep 10 14:57:30.979369 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.978996 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" event={"ID":"7c62e85b-2720-4b6d-9be4-98a2fdfb1031","Type":"ContainerDied","Data":"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4"} Sep 10 14:57:30.979369 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.979056 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9c5df8bf-9qcpv" event={"ID":"7c62e85b-2720-4b6d-9be4-98a2fdfb1031","Type":"ContainerDied","Data":"d8612e56237f050bf307481c78fdd2790f393ed4afcec95cb0787661db56bde7"} Sep 10 14:57:30.979369 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.979078 2581 scope.go:117] "RemoveContainer" containerID="617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4" Sep 10 14:57:30.988206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.988187 2581 scope.go:117] "RemoveContainer" containerID="617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4" Sep 10 14:57:30.988449 ip-10-0-130-182 kubenswrapper[2581]: E0910 14:57:30.988433 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4\": container with ID starting with 617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4 not found: ID does not exist" containerID="617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4" Sep 10 14:57:30.988493 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:30.988458 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4"} err="failed to get container status \"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4\": rpc error: code = NotFound desc = could not find container \"617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4\": container with ID starting with 617c8a123b7aa8a0b22101fe666b47ebccaa51ba49ec431ba8f804e5adb6c7e4 not found: ID does not exist" Sep 10 14:57:31.000341 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:31.000318 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:31.003558 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:31.003532 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b9c5df8bf-9qcpv"] Sep 10 14:57:32.229374 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:32.229342 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" path="/var/lib/kubelet/pods/7c62e85b-2720-4b6d-9be4-98a2fdfb1031/volumes" Sep 10 14:57:48.102803 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:48.102696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:48.191323 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:48.191289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:49.500042 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:49.500008 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:49.600889 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:49.600851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:49.767605 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:49.767501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:49.998925 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:49.998885 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:50.216990 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.216888 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:50.314586 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.314523 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:57:50.340086 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.340051 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nxzvd"] Sep 10 14:57:50.340440 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.340427 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" containerName="manager" Sep 10 14:57:50.340483 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.340442 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" containerName="manager" Sep 10 14:57:50.340516 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.340498 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c62e85b-2720-4b6d-9be4-98a2fdfb1031" containerName="manager" Sep 10 14:57:50.343211 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.343196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.347017 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.346969 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 14:57:50.347173 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.347055 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-npzc9\"" Sep 10 14:57:50.356797 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.356771 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nxzvd"] Sep 10 14:57:50.384487 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.384447 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd"] Sep 10 14:57:50.388360 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.388331 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.402909 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.402870 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd"] Sep 10 14:57:50.490650 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.490615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h65s\" (UniqueName: \"kubernetes.io/projected/ceda0459-fbdc-4ba2-b8be-9235714cf57d-kube-api-access-4h65s\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.490650 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.490652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbff12e-19a2-4b39-a665-74b03e43df4b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.490901 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.490677 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82phn\" (UniqueName: \"kubernetes.io/projected/4dbff12e-19a2-4b39-a665-74b03e43df4b-kube-api-access-82phn\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.490901 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.490845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda0459-fbdc-4ba2-b8be-9235714cf57d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.592256 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.592202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h65s\" (UniqueName: \"kubernetes.io/projected/ceda0459-fbdc-4ba2-b8be-9235714cf57d-kube-api-access-4h65s\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.592256 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.592261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbff12e-19a2-4b39-a665-74b03e43df4b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.592734 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.592288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82phn\" (UniqueName: \"kubernetes.io/projected/4dbff12e-19a2-4b39-a665-74b03e43df4b-kube-api-access-82phn\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.592734 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.592358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda0459-fbdc-4ba2-b8be-9235714cf57d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.593206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.593181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbff12e-19a2-4b39-a665-74b03e43df4b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.593315 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.593292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda0459-fbdc-4ba2-b8be-9235714cf57d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.601150 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.601120 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h65s\" (UniqueName: \"kubernetes.io/projected/ceda0459-fbdc-4ba2-b8be-9235714cf57d-kube-api-access-4h65s\") pod \"payload-processing-5b54b7fbb7-nxzvd\" (UID: \"ceda0459-fbdc-4ba2-b8be-9235714cf57d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.601358 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.601336 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82phn\" (UniqueName: \"kubernetes.io/projected/4dbff12e-19a2-4b39-a665-74b03e43df4b-kube-api-access-82phn\") pod \"payload-pre-processing-59fbc47bf8-fhcsd\" (UID: \"4dbff12e-19a2-4b39-a665-74b03e43df4b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.655359 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.655320 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:50.702903 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.702768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:50.944376 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:50.944272 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nxzvd"] Sep 10 14:57:50.947097 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:57:50.947067 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceda0459_fbdc_4ba2_b8be_9235714cf57d.slice/crio-9dcc901988e000bd38785315d8722195b9a3805872638bb2047466d0b0769f30 WatchSource:0}: Error finding container 9dcc901988e000bd38785315d8722195b9a3805872638bb2047466d0b0769f30: Status 404 returned error can't find the container with id 9dcc901988e000bd38785315d8722195b9a3805872638bb2047466d0b0769f30 Sep 10 14:57:51.003625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:51.003599 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd"] Sep 10 14:57:51.005576 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:57:51.005550 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbff12e_19a2_4b39_a665_74b03e43df4b.slice/crio-f816a3dd98e90057bc9d3639551d5a30ba7af837a87fde3691a812c3b1d176d3 WatchSource:0}: Error finding container f816a3dd98e90057bc9d3639551d5a30ba7af837a87fde3691a812c3b1d176d3: Status 404 returned error can't find the container with id f816a3dd98e90057bc9d3639551d5a30ba7af837a87fde3691a812c3b1d176d3 Sep 10 14:57:51.081362 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:51.081259 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" event={"ID":"4dbff12e-19a2-4b39-a665-74b03e43df4b","Type":"ContainerStarted","Data":"f816a3dd98e90057bc9d3639551d5a30ba7af837a87fde3691a812c3b1d176d3"} Sep 10 14:57:51.082820 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:51.082792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" event={"ID":"ceda0459-fbdc-4ba2-b8be-9235714cf57d","Type":"ContainerStarted","Data":"9dcc901988e000bd38785315d8722195b9a3805872638bb2047466d0b0769f30"} Sep 10 14:57:54.109519 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.109468 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" event={"ID":"4dbff12e-19a2-4b39-a665-74b03e43df4b","Type":"ContainerStarted","Data":"83ed3b3dc559eac4879977c4ceffc57b6fec875913ddea1ce86e51a2d7fb529c"} Sep 10 14:57:54.110076 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.109676 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:57:54.111244 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.111221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" event={"ID":"ceda0459-fbdc-4ba2-b8be-9235714cf57d","Type":"ContainerStarted","Data":"3f7024434734c284ab691236263c83dd731082c91336a731ad61f2d5b60b6408"} Sep 10 14:57:54.111391 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.111342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:57:54.127910 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.127860 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" podStartSLOduration=1.571590637 podStartE2EDuration="4.127845591s" totalImagesPullingTime="2.556254954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:54.125919621 +0000 UTC m=+628.420221753" watchObservedRunningTime="2026-09-10 14:57:54.127845591 +0000 UTC m=+628.422147723" Sep 10 14:57:54.143368 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:54.143309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" podStartSLOduration=1.5227910869999999 podStartE2EDuration="4.143287941s" totalImagesPullingTime="2.620496854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:54.141853104 +0000 UTC m=+628.436155233" watchObservedRunningTime="2026-09-10 14:57:54.143287941 +0000 UTC m=+628.437590074" Sep 10 14:57:59.142854 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:57:59.142813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:00.450553 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:00.450512 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:00.545640 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:00.545610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:05.119967 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:05.119936 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fhcsd" Sep 10 14:58:05.120370 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:05.120146 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-nxzvd" Sep 10 14:58:09.966167 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:09.966126 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:14.447188 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.447147 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86"] Sep 10 14:58:14.451952 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.451928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.455121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.455097 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5vv4r\"" Sep 10 14:58:14.455121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.455107 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 14:58:14.455337 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.455148 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 14:58:14.455337 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.455185 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 14:58:14.463614 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.463592 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86"] Sep 10 14:58:14.486097 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.486056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:14.495939 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.495910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98543b7a-af13-4d3d-9a06-c9820094a9c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.496073 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.495949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.496073 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.495968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjg7\" (UniqueName: \"kubernetes.io/projected/98543b7a-af13-4d3d-9a06-c9820094a9c3-kube-api-access-scjg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.496073 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.496046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.496192 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.496082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.496192 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.496115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.596852 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.596815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597041 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.596872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597041 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.596917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597156 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98543b7a-af13-4d3d-9a06-c9820094a9c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597156 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597262 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjg7\" (UniqueName: \"kubernetes.io/projected/98543b7a-af13-4d3d-9a06-c9820094a9c3-kube-api-access-scjg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597312 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597281 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597397 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597355 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.597571 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.597506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.599414 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.599389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98543b7a-af13-4d3d-9a06-c9820094a9c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.599726 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.599706 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98543b7a-af13-4d3d-9a06-c9820094a9c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.610626 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.610601 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjg7\" (UniqueName: \"kubernetes.io/projected/98543b7a-af13-4d3d-9a06-c9820094a9c3-kube-api-access-scjg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86\" (UID: \"98543b7a-af13-4d3d-9a06-c9820094a9c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:14.762449 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:14.762412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:15.046251 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:15.046221 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86"] Sep 10 14:58:15.049376 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:58:15.049345 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98543b7a_af13_4d3d_9a06_c9820094a9c3.slice/crio-f8389d443b6b3b7603d37092e39d350c60120b8858e429e5fdf24d76491c498c WatchSource:0}: Error finding container f8389d443b6b3b7603d37092e39d350c60120b8858e429e5fdf24d76491c498c: Status 404 returned error can't find the container with id f8389d443b6b3b7603d37092e39d350c60120b8858e429e5fdf24d76491c498c Sep 10 14:58:15.215774 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:15.215699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" event={"ID":"98543b7a-af13-4d3d-9a06-c9820094a9c3","Type":"ContainerStarted","Data":"f8389d443b6b3b7603d37092e39d350c60120b8858e429e5fdf24d76491c498c"} Sep 10 14:58:17.225920 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:17.225879 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" event={"ID":"98543b7a-af13-4d3d-9a06-c9820094a9c3","Type":"ContainerStarted","Data":"a2378e9631f25430dd39dce8e9d55ba042368d221c6b0b35422a6782c25bb7c1"} Sep 10 14:58:17.244565 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:17.244514 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" podStartSLOduration=2.02127388 podStartE2EDuration="3.244496223s" totalImagesPullingTime="1.223222343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:17.243357745 +0000 UTC m=+651.537659874" watchObservedRunningTime="2026-09-10 14:58:17.244496223 +0000 UTC m=+651.538798351" Sep 10 14:58:24.248792 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:24.248740 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:24.763321 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:24.763286 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:24.763578 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:24.763355 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:24.775625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:24.775596 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:25.275992 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:25.275958 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86" Sep 10 14:58:37.475468 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:37.475429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:48.352943 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:48.352903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:53.534326 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:53.534293 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:54.232169 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.232128 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:54.343615 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.343580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:54.495138 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.495100 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz"] Sep 10 14:58:54.497674 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.497650 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.500941 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.500918 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 14:58:54.517957 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.517926 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz"] Sep 10 14:58:54.646822 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.646750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.647214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.646835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13929071-919f-4bb6-9498-d3bf7c8764a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.647214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.646883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvqz\" (UniqueName: \"kubernetes.io/projected/13929071-919f-4bb6-9498-d3bf7c8764a3-kube-api-access-mgvqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.647214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.646944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.647214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.646964 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.647214 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.647019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.648955 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.648917 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:58:54.748317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13929071-919f-4bb6-9498-d3bf7c8764a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvqz\" (UniqueName: \"kubernetes.io/projected/13929071-919f-4bb6-9498-d3bf7c8764a3-kube-api-access-mgvqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748317 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748687 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748687 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748846 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748823 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748907 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.748907 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.748893 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.750819 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.750795 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13929071-919f-4bb6-9498-d3bf7c8764a3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.751218 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.751198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13929071-919f-4bb6-9498-d3bf7c8764a3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.758845 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.758820 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvqz\" (UniqueName: \"kubernetes.io/projected/13929071-919f-4bb6-9498-d3bf7c8764a3-kube-api-access-mgvqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2cqpz\" (UID: \"13929071-919f-4bb6-9498-d3bf7c8764a3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:54.808622 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:54.808547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:58:55.092471 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:55.092445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz"] Sep 10 14:58:55.094775 ip-10-0-130-182 kubenswrapper[2581]: W0910 14:58:55.094712 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13929071_919f_4bb6_9498_d3bf7c8764a3.slice/crio-4ae1786dfd60d02e34ac6c5e7b9502888c75861e59151212621b28832995c3b1 WatchSource:0}: Error finding container 4ae1786dfd60d02e34ac6c5e7b9502888c75861e59151212621b28832995c3b1: Status 404 returned error can't find the container with id 4ae1786dfd60d02e34ac6c5e7b9502888c75861e59151212621b28832995c3b1 Sep 10 14:58:55.096452 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:55.096431 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:58:55.407625 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:55.407581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" event={"ID":"13929071-919f-4bb6-9498-d3bf7c8764a3","Type":"ContainerStarted","Data":"a30a5897d5383c4005ae4dc1a382dcd0e032376cc5c49e93420f972fa110bfaf"} Sep 10 14:58:55.407926 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:55.407656 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" event={"ID":"13929071-919f-4bb6-9498-d3bf7c8764a3","Type":"ContainerStarted","Data":"4ae1786dfd60d02e34ac6c5e7b9502888c75861e59151212621b28832995c3b1"} Sep 10 14:58:55.454598 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:58:55.454539 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" podStartSLOduration=1.24804795 podStartE2EDuration="1.454519741s" totalImagesPullingTime="206.471791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:55.452509827 +0000 UTC m=+689.746811960" watchObservedRunningTime="2026-09-10 14:58:55.454519741 +0000 UTC m=+689.748821869" Sep 10 14:59:02.810104 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:02.810066 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:02.951603 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:02.951562 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:04.809206 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:04.809161 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:59:04.809612 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:04.809230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:59:04.821589 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:04.821564 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:59:05.469735 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:05.469706 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2cqpz" Sep 10 14:59:13.571291 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:13.571253 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:13.732121 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:13.732082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:23.708158 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:23.708080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:30.313740 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:30.313704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:35.910375 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:35.910333 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 14:59:41.203813 ip-10-0-130-182 kubenswrapper[2581]: I0910 14:59:41.203777 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:00:00.183271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.183236 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl"] Sep 10 15:00:00.185664 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.185648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" Sep 10 15:00:00.191680 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.191658 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:00:00.194998 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.194980 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" Sep 10 15:00:00.211571 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.211539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl"] Sep 10 15:00:00.484871 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.484777 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl"] Sep 10 15:00:00.487135 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:00:00.487109 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45de12e_2b9c_4375_98ed_cc8438c2f36d.slice/crio-c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114 WatchSource:0}: Error finding container c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114: Status 404 returned error can't find the container with id c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114 Sep 10 15:00:00.727624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:00.727580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" event={"ID":"c45de12e-2b9c-4375-98ed-cc8438c2f36d","Type":"ContainerStarted","Data":"c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114"} Sep 10 15:00:03.743859 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:03.743814 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" event={"ID":"c45de12e-2b9c-4375-98ed-cc8438c2f36d","Type":"ContainerStarted","Data":"511a290fff779df3157cf0c19239e0ab3ddf7ae069294e82a9d8139adbc05eb2"} Sep 10 15:00:03.766689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:03.766638 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" podStartSLOduration=1.177149897 podStartE2EDuration="3.766598443s" totalImagesPullingTime="2.589448546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:03.766311962 +0000 UTC m=+758.060614092" watchObservedRunningTime="2026-09-10 15:00:03.766598443 +0000 UTC m=+758.060900571" Sep 10 15:00:04.750584 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:04.750547 2581 generic.go:356] "Generic (PLEG): container finished" podID="c45de12e-2b9c-4375-98ed-cc8438c2f36d" containerID="511a290fff779df3157cf0c19239e0ab3ddf7ae069294e82a9d8139adbc05eb2" exitCode=0 Sep 10 15:00:04.751051 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:04.750636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" event={"ID":"c45de12e-2b9c-4375-98ed-cc8438c2f36d","Type":"ContainerDied","Data":"511a290fff779df3157cf0c19239e0ab3ddf7ae069294e82a9d8139adbc05eb2"} Sep 10 15:00:05.891909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:05.891883 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" Sep 10 15:00:06.761126 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:06.761087 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" event={"ID":"c45de12e-2b9c-4375-98ed-cc8438c2f36d","Type":"ContainerDied","Data":"c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114"} Sep 10 15:00:06.761126 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:06.761121 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817540-s8gpl" Sep 10 15:00:06.761326 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:06.761141 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46a0eb302fb41c4998ac263eea5d547c73ca25fdb5610281644b1531e656114" Sep 10 15:00:32.457965 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:32.457928 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:00:32.728065 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:00:32.727966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:04.013111 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.013075 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-559f9485b-m8q7w"] Sep 10 15:01:04.013580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.013480 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45de12e-2b9c-4375-98ed-cc8438c2f36d" containerName="cleanup" Sep 10 15:01:04.013580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.013494 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45de12e-2b9c-4375-98ed-cc8438c2f36d" containerName="cleanup" Sep 10 15:01:04.013580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.013579 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45de12e-2b9c-4375-98ed-cc8438c2f36d" containerName="cleanup" Sep 10 15:01:04.015627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.015608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.027943 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.027907 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-559f9485b-m8q7w"] Sep 10 15:01:04.043178 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.043143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz7d\" (UniqueName: \"kubernetes.io/projected/4bfa5a1e-afc4-4161-9113-6b367d546315-kube-api-access-4hz7d\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.043354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.043214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4bfa5a1e-afc4-4161-9113-6b367d546315-tls-cert\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.144013 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.143975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz7d\" (UniqueName: \"kubernetes.io/projected/4bfa5a1e-afc4-4161-9113-6b367d546315-kube-api-access-4hz7d\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.144188 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.144038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4bfa5a1e-afc4-4161-9113-6b367d546315-tls-cert\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.146727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.146697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4bfa5a1e-afc4-4161-9113-6b367d546315-tls-cert\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.155855 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.155825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz7d\" (UniqueName: \"kubernetes.io/projected/4bfa5a1e-afc4-4161-9113-6b367d546315-kube-api-access-4hz7d\") pod \"authorino-559f9485b-m8q7w\" (UID: \"4bfa5a1e-afc4-4161-9113-6b367d546315\") " pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.324685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.324601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-559f9485b-m8q7w" Sep 10 15:01:04.607151 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:04.607125 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-559f9485b-m8q7w"] Sep 10 15:01:04.609749 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:04.609721 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bfa5a1e_afc4_4161_9113_6b367d546315.slice/crio-cddb192ed4e9f1f6365c10968ed2d7ccf8090a798177ecb2a1b0a40c1fb62916 WatchSource:0}: Error finding container cddb192ed4e9f1f6365c10968ed2d7ccf8090a798177ecb2a1b0a40c1fb62916: Status 404 returned error can't find the container with id cddb192ed4e9f1f6365c10968ed2d7ccf8090a798177ecb2a1b0a40c1fb62916 Sep 10 15:01:05.058500 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:05.058462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-559f9485b-m8q7w" event={"ID":"4bfa5a1e-afc4-4161-9113-6b367d546315","Type":"ContainerStarted","Data":"cddb192ed4e9f1f6365c10968ed2d7ccf8090a798177ecb2a1b0a40c1fb62916"} Sep 10 15:01:05.158437 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:05.158402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:05.258946 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:05.258912 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:06.067255 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.067222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-559f9485b-m8q7w" event={"ID":"4bfa5a1e-afc4-4161-9113-6b367d546315","Type":"ContainerStarted","Data":"b072dd3824648e591f057fe54397f0ab967d88e26fd53d799eb56ef9f46789e1"} Sep 10 15:01:06.108243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.108169 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-559f9485b-m8q7w" podStartSLOduration=2.425842035 podStartE2EDuration="3.108148219s" totalImagesPullingTime="682.306184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:06.10682795 +0000 UTC m=+820.401130080" watchObservedRunningTime="2026-09-10 15:01:06.108148219 +0000 UTC m=+820.402450350" Sep 10 15:01:06.174116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.173795 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 15:01:06.174116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.174031 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-94769c5bf-skr8s" podUID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" containerName="authorino" containerID="cri-o://18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68" gracePeriod=30 Sep 10 15:01:06.458919 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.458885 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 15:01:06.566931 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.566883 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert\") pod \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " Sep 10 15:01:06.567135 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.566952 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt8x4\" (UniqueName: \"kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4\") pod \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\" (UID: \"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d\") " Sep 10 15:01:06.569110 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.569074 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4" (OuterVolumeSpecName: "kube-api-access-bt8x4") pod "ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" (UID: "ef9a33bf-d252-4367-8ba4-5c446bc7ec3d"). InnerVolumeSpecName "kube-api-access-bt8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:01:06.578910 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.578868 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" (UID: "ef9a33bf-d252-4367-8ba4-5c446bc7ec3d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:01:06.668057 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.667963 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt8x4\" (UniqueName: \"kubernetes.io/projected/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-kube-api-access-bt8x4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:01:06.668057 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.667994 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d-tls-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:01:06.755641 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.755606 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:06.959219 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:06.959111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:07.074233 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.074202 2581 generic.go:356] "Generic (PLEG): container finished" podID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" containerID="18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68" exitCode=0 Sep 10 15:01:07.074716 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.074248 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-94769c5bf-skr8s" Sep 10 15:01:07.074716 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.074295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94769c5bf-skr8s" event={"ID":"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d","Type":"ContainerDied","Data":"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68"} Sep 10 15:01:07.074716 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.074348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-94769c5bf-skr8s" event={"ID":"ef9a33bf-d252-4367-8ba4-5c446bc7ec3d","Type":"ContainerDied","Data":"168e9f64a431ca99a57c4dde9c937026b6a038c42094d5551e98ec33be89e0b4"} Sep 10 15:01:07.074716 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.074371 2581 scope.go:117] "RemoveContainer" containerID="18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68" Sep 10 15:01:07.087250 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.087227 2581 scope.go:117] "RemoveContainer" containerID="18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68" Sep 10 15:01:07.087650 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:01:07.087628 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68\": container with ID starting with 18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68 not found: ID does not exist" containerID="18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68" Sep 10 15:01:07.087792 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.087664 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68"} err="failed to get container status \"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68\": rpc error: code = NotFound desc = could not find container \"18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68\": container with ID starting with 18791dfae24eb0468fd075a33a94d4a3d4da3999a83a1c47de560ebadad89f68 not found: ID does not exist" Sep 10 15:01:07.117712 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.117679 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 15:01:07.122955 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.122925 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-94769c5bf-skr8s"] Sep 10 15:01:07.812243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.812177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:07.905101 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:07.905056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:08.230540 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:08.230453 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" path="/var/lib/kubelet/pods/ef9a33bf-d252-4367-8ba4-5c446bc7ec3d/volumes" Sep 10 15:01:25.314923 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:25.314885 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:25.440901 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:25.440863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:25.858015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:25.857980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:26.002644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:26.002600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:26.821031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:26.820973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:26.913130 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:26.913097 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:27.321726 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.321693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:27.413636 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.413602 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7555-e006be"] Sep 10 15:01:27.414079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.414060 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" containerName="authorino" Sep 10 15:01:27.414175 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.414082 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" containerName="authorino" Sep 10 15:01:27.414227 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.414204 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9a33bf-d252-4367-8ba4-5c446bc7ec3d" containerName="authorino" Sep 10 15:01:27.417047 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.417024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:27.419532 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.419510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlvr\"" Sep 10 15:01:27.424670 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.424365 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7555-e006be"] Sep 10 15:01:27.561462 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.561422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmrj\" (UniqueName: \"kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj\") pod \"test-curl-7555-e006be\" (UID: \"bc65901f-25a0-4cd1-b240-0cbea59c604d\") " pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:27.662275 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.662160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmrj\" (UniqueName: \"kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj\") pod \"test-curl-7555-e006be\" (UID: \"bc65901f-25a0-4cd1-b240-0cbea59c604d\") " pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:27.671222 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.671193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmrj\" (UniqueName: \"kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj\") pod \"test-curl-7555-e006be\" (UID: \"bc65901f-25a0-4cd1-b240-0cbea59c604d\") " pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:27.727412 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.727374 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:27.998607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:27.998486 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7555-e006be"] Sep 10 15:01:28.001290 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:28.001256 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc65901f_25a0_4cd1_b240_0cbea59c604d.slice/crio-0ef038b46e2de9bb41ce1d4867250d2e45be218fabfc9354ac082a6d83b2034c WatchSource:0}: Error finding container 0ef038b46e2de9bb41ce1d4867250d2e45be218fabfc9354ac082a6d83b2034c: Status 404 returned error can't find the container with id 0ef038b46e2de9bb41ce1d4867250d2e45be218fabfc9354ac082a6d83b2034c Sep 10 15:01:28.191708 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:28.191664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7555-e006be" event={"ID":"bc65901f-25a0-4cd1-b240-0cbea59c604d","Type":"ContainerStarted","Data":"0ef038b46e2de9bb41ce1d4867250d2e45be218fabfc9354ac082a6d83b2034c"} Sep 10 15:01:28.481050 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:28.480969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:28.606136 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:28.606101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:29.909219 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:29.909184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:30.014995 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.014959 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:30.484749 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.484709 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:30.582809 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.582778 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:30.651266 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.651228 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:01:30.654036 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.654012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.655455 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.655435 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:01:30.657429 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.657404 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e577f0\"" Sep 10 15:01:30.657916 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.657898 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.660014 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.659994 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e577f0-dockercfg-9v7w8\"" Sep 10 15:01:30.668877 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.668841 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:01:30.675247 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.675224 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:01:30.683335 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.683308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:30.789713 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.789679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqw7\" (UniqueName: \"kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.789920 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.789736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.789920 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.789866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.790001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.789924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhcp\" (UniqueName: \"kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.890862 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.890827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.891039 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.890882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhcp\" (UniqueName: \"kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.891039 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.890914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqw7\" (UniqueName: \"kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.891039 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.890951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.891649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.891627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.891697 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.891632 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.901572 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.901528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhcp\" (UniqueName: \"kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp\") pod \"payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.902220 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.902193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqw7\" (UniqueName: \"kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7\") pod \"payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:30.966350 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.966306 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:30.973881 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:30.973845 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:31.214654 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.214615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7555-e006be" event={"ID":"bc65901f-25a0-4cd1-b240-0cbea59c604d","Type":"ContainerStarted","Data":"7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a"} Sep 10 15:01:31.230621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.230569 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7555-e006be" podStartSLOduration=1.724882045 podStartE2EDuration="4.230550319s" totalImagesPullingTime="2.505668274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:31.229621776 +0000 UTC m=+845.523923906" watchObservedRunningTime="2026-09-10 15:01:31.230550319 +0000 UTC m=+845.524852448" Sep 10 15:01:31.248541 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.248507 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:01:31.249643 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:31.249618 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb8fbf8_4844_4865_867a_d9441134689e.slice/crio-5872b120a209669ecab338119f3e81c4bf98839b8b3d5acbd2cd57b6e6e1234a WatchSource:0}: Error finding container 5872b120a209669ecab338119f3e81c4bf98839b8b3d5acbd2cd57b6e6e1234a: Status 404 returned error can't find the container with id 5872b120a209669ecab338119f3e81c4bf98839b8b3d5acbd2cd57b6e6e1234a Sep 10 15:01:31.261183 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.261160 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:01:31.264167 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:31.264137 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6759748b_e2af_4e03_b648_809fdabc2e84.slice/crio-55ea4afa89bbeb8fd2a37baa505f636e675531d83efdb5049db1a24dcd33a05a WatchSource:0}: Error finding container 55ea4afa89bbeb8fd2a37baa505f636e675531d83efdb5049db1a24dcd33a05a: Status 404 returned error can't find the container with id 55ea4afa89bbeb8fd2a37baa505f636e675531d83efdb5049db1a24dcd33a05a Sep 10 15:01:31.850584 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.850548 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7555-e006be"] Sep 10 15:01:31.857611 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:31.857577 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7555-e006be"] Sep 10 15:01:32.222969 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.222926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" event={"ID":"deb8fbf8-4844-4865-867a-d9441134689e","Type":"ContainerStarted","Data":"3b0e55ce5bff608aa63f5ab0508f7197bb96285877054b2efb465477c8d6f512"} Sep 10 15:01:32.223462 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.223006 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" event={"ID":"deb8fbf8-4844-4865-867a-d9441134689e","Type":"ContainerStarted","Data":"5872b120a209669ecab338119f3e81c4bf98839b8b3d5acbd2cd57b6e6e1234a"} Sep 10 15:01:32.223462 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.223035 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:32.225164 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:01:32.225134 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7555-e006be" container="test-curl-7555-e006be" Sep 10 15:01:32.230567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.230540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" event={"ID":"6759748b-e2af-4e03-b648-809fdabc2e84","Type":"ContainerStarted","Data":"345e515d9791754d162188cbb0c5c7f33f56f01476304955c8018dbf0a53f1eb"} Sep 10 15:01:32.230700 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.230588 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" event={"ID":"6759748b-e2af-4e03-b648-809fdabc2e84","Type":"ContainerStarted","Data":"55ea4afa89bbeb8fd2a37baa505f636e675531d83efdb5049db1a24dcd33a05a"} Sep 10 15:01:32.240746 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.240691 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" podStartSLOduration=2.240677969 podStartE2EDuration="2.240677969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:32.238653907 +0000 UTC m=+846.532956036" watchObservedRunningTime="2026-09-10 15:01:32.240677969 +0000 UTC m=+846.534980098" Sep 10 15:01:32.254112 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:32.254045 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" podStartSLOduration=2.254025682 podStartE2EDuration="2.254025682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:32.253696468 +0000 UTC m=+846.547998598" watchObservedRunningTime="2026-09-10 15:01:32.254025682 +0000 UTC m=+846.548327812" Sep 10 15:01:33.232233 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:33.232192 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:33.232732 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:33.232192 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7555-e006be" podUID="bc65901f-25a0-4cd1-b240-0cbea59c604d" containerName="test-curl-7555-e006be" containerID="cri-o://7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a" gracePeriod=2 Sep 10 15:01:33.409209 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:33.409173 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:33.474223 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:33.474185 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:33.647284 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:33.647251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:34.080437 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:34.080399 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:34.302438 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:34.302399 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:35.427381 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:35.427357 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:35.535089 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:35.535057 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvmrj\" (UniqueName: \"kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj\") pod \"bc65901f-25a0-4cd1-b240-0cbea59c604d\" (UID: \"bc65901f-25a0-4cd1-b240-0cbea59c604d\") " Sep 10 15:01:35.537243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:35.537215 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj" (OuterVolumeSpecName: "kube-api-access-jvmrj") pod "bc65901f-25a0-4cd1-b240-0cbea59c604d" (UID: "bc65901f-25a0-4cd1-b240-0cbea59c604d"). InnerVolumeSpecName "kube-api-access-jvmrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:01:35.635947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:35.635908 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmrj\" (UniqueName: \"kubernetes.io/projected/bc65901f-25a0-4cd1-b240-0cbea59c604d-kube-api-access-jvmrj\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:01:36.096864 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.096829 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:36.230881 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.230850 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc65901f-25a0-4cd1-b240-0cbea59c604d" path="/var/lib/kubelet/pods/bc65901f-25a0-4cd1-b240-0cbea59c604d/volumes" Sep 10 15:01:36.248264 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.248235 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc65901f-25a0-4cd1-b240-0cbea59c604d" containerID="7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a" exitCode=137 Sep 10 15:01:36.248412 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.248280 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-e006be" Sep 10 15:01:36.248412 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.248332 2581 scope.go:117] "RemoveContainer" containerID="7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a" Sep 10 15:01:36.256797 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.256778 2581 scope.go:117] "RemoveContainer" containerID="7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a" Sep 10 15:01:36.257075 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:01:36.257058 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a\": container with ID starting with 7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a not found: ID does not exist" containerID="7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a" Sep 10 15:01:36.257116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.257088 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a"} err="failed to get container status \"7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a\": rpc error: code = NotFound desc = could not find container \"7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a\": container with ID starting with 7a2f04741984348ebf3a327682af6e69b602e1570337ca1d11c8070a4058556a not found: ID does not exist" Sep 10 15:01:36.422853 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.422742 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:36.619351 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.619304 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:36.882183 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:36.882129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:37.013115 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:37.013076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.007879 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.007825 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.106451 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.106411 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.464465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.464380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.581307 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.581272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.687836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.687798 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:01:38.688271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.688254 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc65901f-25a0-4cd1-b240-0cbea59c604d" containerName="test-curl-7555-e006be" Sep 10 15:01:38.688360 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.688273 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc65901f-25a0-4cd1-b240-0cbea59c604d" containerName="test-curl-7555-e006be" Sep 10 15:01:38.688423 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.688404 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc65901f-25a0-4cd1-b240-0cbea59c604d" containerName="test-curl-7555-e006be" Sep 10 15:01:38.691242 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.691221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.694031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.694011 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-24e113dd-dockercfg-8phts\"" Sep 10 15:01:38.694134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.694068 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-24e113dd\"" Sep 10 15:01:38.706071 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.706042 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:01:38.713792 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.713747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:38.763386 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.763346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.763577 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.763409 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnbk\" (UniqueName: \"kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.818003 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.817974 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:01:38.820340 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.820324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.836421 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.836393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:01:38.864104 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.864067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnbk\" (UniqueName: \"kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.864387 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.864365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.864570 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.864555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm8n\" (UniqueName: \"kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.864724 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.864698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.865558 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.865532 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.875102 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.875078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnbk\" (UniqueName: \"kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk\") pod \"payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:38.965691 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.965655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.965957 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.965726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm8n\" (UniqueName: \"kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.966390 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.966366 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:38.977971 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:38.977943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm8n\" (UniqueName: \"kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n\") pod \"payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:39.003683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.003654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:39.130508 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.130473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:39.276200 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.275999 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:01:39.278734 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:39.278702 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404fc626_a7d8_4986_97fe_8d9b42209121.slice/crio-e323984e335eae7f3e900996ee0286988d6569098fdf31ab3aa53c5d190e82be WatchSource:0}: Error finding container e323984e335eae7f3e900996ee0286988d6569098fdf31ab3aa53c5d190e82be: Status 404 returned error can't find the container with id e323984e335eae7f3e900996ee0286988d6569098fdf31ab3aa53c5d190e82be Sep 10 15:01:39.401086 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.401014 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:01:39.403919 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:39.403894 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02382b30_a4fe_4eee_b624_8892f2d6e593.slice/crio-1d20dec66c796cdf531665f23750e987a3c25803cb3ea9a83ebf75b1a0533da9 WatchSource:0}: Error finding container 1d20dec66c796cdf531665f23750e987a3c25803cb3ea9a83ebf75b1a0533da9: Status 404 returned error can't find the container with id 1d20dec66c796cdf531665f23750e987a3c25803cb3ea9a83ebf75b1a0533da9 Sep 10 15:01:39.532120 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.532028 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:39.647134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:39.647107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:40.272626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.272588 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" event={"ID":"404fc626-a7d8-4986-97fe-8d9b42209121","Type":"ContainerStarted","Data":"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea"} Sep 10 15:01:40.273106 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.272783 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:40.273106 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.272836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" event={"ID":"404fc626-a7d8-4986-97fe-8d9b42209121","Type":"ContainerStarted","Data":"e323984e335eae7f3e900996ee0286988d6569098fdf31ab3aa53c5d190e82be"} Sep 10 15:01:40.274247 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.274220 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" event={"ID":"02382b30-a4fe-4eee-b624-8892f2d6e593","Type":"ContainerStarted","Data":"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35"} Sep 10 15:01:40.274373 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.274265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" event={"ID":"02382b30-a4fe-4eee-b624-8892f2d6e593","Type":"ContainerStarted","Data":"1d20dec66c796cdf531665f23750e987a3c25803cb3ea9a83ebf75b1a0533da9"} Sep 10 15:01:40.274373 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.274319 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:40.289021 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.288974 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" podStartSLOduration=2.288958949 podStartE2EDuration="2.288958949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:40.288338301 +0000 UTC m=+854.582640454" watchObservedRunningTime="2026-09-10 15:01:40.288958949 +0000 UTC m=+854.583261078" Sep 10 15:01:40.306099 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.306044 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" podStartSLOduration=2.306029076 podStartE2EDuration="2.306029076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:40.30432485 +0000 UTC m=+854.598626980" watchObservedRunningTime="2026-09-10 15:01:40.306029076 +0000 UTC m=+854.600331205" Sep 10 15:01:40.912940 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:40.912901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:41.042811 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.042746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:41.174197 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.174113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:41.626928 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.626888 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:41.728807 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.728776 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:41.762809 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.762773 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:01:41.766445 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.766427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.772470 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.772442 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c75e68\"" Sep 10 15:01:41.785432 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.785401 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:01:41.793206 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.793151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62cq\" (UniqueName: \"kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.793420 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.793403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.822963 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.822931 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:01:41.825571 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.825548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:41.840282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.840260 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c75e68-dockercfg-5b8sr\"" Sep 10 15:01:41.851168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.851104 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:01:41.895687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.894693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62cq\" (UniqueName: \"kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.895687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.894747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.895687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.894826 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:41.895687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.894954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pcs\" (UniqueName: \"kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:41.896134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.896110 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.905204 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.905178 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62cq\" (UniqueName: \"kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq\") pod \"payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:41.996506 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.996459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pcs\" (UniqueName: \"kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:41.996706 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.996544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:41.997312 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:41.997289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:42.010238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:42.010200 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pcs\" (UniqueName: \"kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs\") pod \"payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:42.077915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:42.077878 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:42.136375 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:42.135897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:42.365316 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:42.365280 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:01:42.368875 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:42.368828 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30ffa16_cf70_49fe_8c3f_930e95a4f0c5.slice/crio-f03c9bad890cbd8fd51500c5aa020317b375c8d23070f538b9f38d71f60de8dd WatchSource:0}: Error finding container f03c9bad890cbd8fd51500c5aa020317b375c8d23070f538b9f38d71f60de8dd: Status 404 returned error can't find the container with id f03c9bad890cbd8fd51500c5aa020317b375c8d23070f538b9f38d71f60de8dd Sep 10 15:01:42.423003 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:42.422973 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:01:42.426019 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:01:42.425991 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4683ba2b_3433_475c_a17f_2a31930df65a.slice/crio-00eeddc804a0e8d5fac0de0a4f50ccf0a0508d370447cbe4bb042ba1c1839006 WatchSource:0}: Error finding container 00eeddc804a0e8d5fac0de0a4f50ccf0a0508d370447cbe4bb042ba1c1839006: Status 404 returned error can't find the container with id 00eeddc804a0e8d5fac0de0a4f50ccf0a0508d370447cbe4bb042ba1c1839006 Sep 10 15:01:43.234512 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.234479 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:01:43.293624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.293580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" event={"ID":"4683ba2b-3433-475c-a17f-2a31930df65a","Type":"ContainerStarted","Data":"f6cba88a6ed6973a38b6ecf91e9e516e512e2e804c7a3fc918849848099c19af"} Sep 10 15:01:43.293816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.293645 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" event={"ID":"4683ba2b-3433-475c-a17f-2a31930df65a","Type":"ContainerStarted","Data":"00eeddc804a0e8d5fac0de0a4f50ccf0a0508d370447cbe4bb042ba1c1839006"} Sep 10 15:01:43.294509 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.294441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:43.296615 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.296590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" event={"ID":"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5","Type":"ContainerStarted","Data":"f8115fe6930d68ad2e2b2e8d4936b11b48625c6b3ed5c6befda179b43e442ca0"} Sep 10 15:01:43.296738 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.296646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" event={"ID":"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5","Type":"ContainerStarted","Data":"f03c9bad890cbd8fd51500c5aa020317b375c8d23070f538b9f38d71f60de8dd"} Sep 10 15:01:43.296738 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.296716 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:43.313824 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.313773 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" podStartSLOduration=2.313737839 podStartE2EDuration="2.313737839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:43.311732553 +0000 UTC m=+857.606034683" watchObservedRunningTime="2026-09-10 15:01:43.313737839 +0000 UTC m=+857.608039958" Sep 10 15:01:43.332511 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:43.332460 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" podStartSLOduration=2.332441021 podStartE2EDuration="2.332441021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:43.33230641 +0000 UTC m=+857.626608542" watchObservedRunningTime="2026-09-10 15:01:43.332441021 +0000 UTC m=+857.626743141" Sep 10 15:01:44.239831 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:44.239802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:01:45.020180 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:45.020139 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:45.253830 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:45.253793 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:45.359936 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:45.359844 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:45.459082 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:45.459031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:48.188859 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:48.188822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:50.124665 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:50.124616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:50.985020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:50.984985 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:51.087205 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:51.087166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:51.226499 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:51.226464 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:51.281435 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:51.281400 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:01:51.281629 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:51.281490 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:01:52.249583 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:52.249545 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:53.454941 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:53.454900 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:53.796499 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:53.796459 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:54.179136 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:54.179044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:54.305165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:54.305133 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:01:54.601491 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:54.601458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:55.312676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:55.312632 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:01:57.697513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:57.697471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:57.801219 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:57.801180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:57.901407 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:57.901376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:58.003729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:58.003691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:58.102048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:58.102016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:58.217261 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:58.217225 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:59.046210 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.046169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:59.162309 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.162271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:59.749054 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.749021 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:59.847969 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.847937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:01:59.940667 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.940630 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:01:59.943305 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.943289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:01:59.946074 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.946053 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6356e4-dockercfg-zfwjk\"" Sep 10 15:01:59.946208 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.946053 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6356e4\"" Sep 10 15:01:59.956375 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:01:59.956347 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:02:00.021366 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.021281 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:02:00.024250 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.024222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.035658 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.035628 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:02:00.085734 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.085690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.086111 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.085771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72t7\" (UniqueName: \"kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.187141 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.187106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stvp\" (UniqueName: \"kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.187353 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.187157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72t7\" (UniqueName: \"kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.187353 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.187235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.187467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.187353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.188116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.188095 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.198839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.198811 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72t7\" (UniqueName: \"kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7\") pod \"payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.254366 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.254297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:00.288832 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.288711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stvp\" (UniqueName: \"kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.289020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.288893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.289552 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.289530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.320844 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.320603 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stvp\" (UniqueName: \"kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp\") pod \"payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.337398 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.337368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:00.541561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.541530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:02:00.542103 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:00.542079 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66af9654_f0d0_4200_a47c_82594e91c352.slice/crio-097efd98cc7799313891d6f612e4b359e79b879c022ad805e9a8b6aafc6f3cb9 WatchSource:0}: Error finding container 097efd98cc7799313891d6f612e4b359e79b879c022ad805e9a8b6aafc6f3cb9: Status 404 returned error can't find the container with id 097efd98cc7799313891d6f612e4b359e79b879c022ad805e9a8b6aafc6f3cb9 Sep 10 15:02:00.627923 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:00.627898 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:02:00.631129 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:00.631099 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af0ecce_73dc_4ca6_a272_67729798eb1b.slice/crio-12d410febb15f09a5970cae1adc5f309e5dc28acbb49533578bc0c5e3720a0cd WatchSource:0}: Error finding container 12d410febb15f09a5970cae1adc5f309e5dc28acbb49533578bc0c5e3720a0cd: Status 404 returned error can't find the container with id 12d410febb15f09a5970cae1adc5f309e5dc28acbb49533578bc0c5e3720a0cd Sep 10 15:02:01.229609 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.229569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:01.404839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.404792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" event={"ID":"66af9654-f0d0-4200-a47c-82594e91c352","Type":"ContainerStarted","Data":"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954"} Sep 10 15:02:01.405041 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.404873 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:01.405041 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.404905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" event={"ID":"66af9654-f0d0-4200-a47c-82594e91c352","Type":"ContainerStarted","Data":"097efd98cc7799313891d6f612e4b359e79b879c022ad805e9a8b6aafc6f3cb9"} Sep 10 15:02:01.406605 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.406575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" event={"ID":"1af0ecce-73dc-4ca6-a272-67729798eb1b","Type":"ContainerStarted","Data":"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f"} Sep 10 15:02:01.406727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.406622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" event={"ID":"1af0ecce-73dc-4ca6-a272-67729798eb1b","Type":"ContainerStarted","Data":"12d410febb15f09a5970cae1adc5f309e5dc28acbb49533578bc0c5e3720a0cd"} Sep 10 15:02:01.406727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.406647 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:01.425636 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.425581 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" podStartSLOduration=2.425565772 podStartE2EDuration="2.425565772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:01.423235295 +0000 UTC m=+875.717537438" watchObservedRunningTime="2026-09-10 15:02:01.425565772 +0000 UTC m=+875.719867901" Sep 10 15:02:01.444183 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:01.444121 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" podStartSLOduration=2.444098655 podStartE2EDuration="2.444098655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:01.44251799 +0000 UTC m=+875.736820120" watchObservedRunningTime="2026-09-10 15:02:01.444098655 +0000 UTC m=+875.738400785" Sep 10 15:02:02.083585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.083552 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:02.437306 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.437223 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d"] Sep 10 15:02:02.442354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.442328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:02.444871 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.444846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:02:02.449349 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.449321 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d"] Sep 10 15:02:02.610404 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.610369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24e113dd-rp62d\" (UID: \"94f8612b-b85a-456f-b13f-cc9f1447e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:02.711870 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.711731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24e113dd-rp62d\" (UID: \"94f8612b-b85a-456f-b13f-cc9f1447e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:02.712403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.712381 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24e113dd-rp62d\" (UID: \"94f8612b-b85a-456f-b13f-cc9f1447e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:02.757711 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.757667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:02.886972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:02.886941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:03.042094 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:03.042067 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d"] Sep 10 15:02:03.043499 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:03.043472 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f8612b_b85a_456f_b13f_cc9f1447e0ef.slice/crio-950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde WatchSource:0}: Error finding container 950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde: Status 404 returned error can't find the container with id 950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde Sep 10 15:02:03.425207 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:03.425162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" event={"ID":"94f8612b-b85a-456f-b13f-cc9f1447e0ef","Type":"ContainerStarted","Data":"93dc7e51c93208377eaba56be608dbbce7befc0c4077dd806f451c413ab71a0d"} Sep 10 15:02:03.425383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:03.425242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" event={"ID":"94f8612b-b85a-456f-b13f-cc9f1447e0ef","Type":"ContainerStarted","Data":"950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde"} Sep 10 15:02:03.448713 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:03.448649 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" podStartSLOduration=1.448633712 podStartE2EDuration="1.448633712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:03.446112757 +0000 UTC m=+877.740414916" watchObservedRunningTime="2026-09-10 15:02:03.448633712 +0000 UTC m=+877.742935897" Sep 10 15:02:03.730156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:03.730060 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:04.432211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.432177 2581 generic.go:356] "Generic (PLEG): container finished" podID="94f8612b-b85a-456f-b13f-cc9f1447e0ef" containerID="93dc7e51c93208377eaba56be608dbbce7befc0c4077dd806f451c413ab71a0d" exitCode=0 Sep 10 15:02:04.432422 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.432262 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" event={"ID":"94f8612b-b85a-456f-b13f-cc9f1447e0ef","Type":"ContainerDied","Data":"93dc7e51c93208377eaba56be608dbbce7befc0c4077dd806f451c413ab71a0d"} Sep 10 15:02:04.474064 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.474029 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:04.582031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.582001 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:04.778489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.778453 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:04.880660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:04.880602 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:05.577295 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:05.577263 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:05.740993 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:05.740959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca\") pod \"94f8612b-b85a-456f-b13f-cc9f1447e0ef\" (UID: \"94f8612b-b85a-456f-b13f-cc9f1447e0ef\") " Sep 10 15:02:05.741296 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:05.741271 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94f8612b-b85a-456f-b13f-cc9f1447e0ef" (UID: "94f8612b-b85a-456f-b13f-cc9f1447e0ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:05.842543 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:05.842505 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8612b-b85a-456f-b13f-cc9f1447e0ef-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:06.445741 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:06.445706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" event={"ID":"94f8612b-b85a-456f-b13f-cc9f1447e0ef","Type":"ContainerDied","Data":"950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde"} Sep 10 15:02:06.445970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:06.445790 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950b87b958ba3ded52186541a55f2f3770d8081a45c2e63088c035cef8172dde" Sep 10 15:02:06.445970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:06.445713 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d" Sep 10 15:02:08.244882 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.244846 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:08.906933 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.906899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:08.949768 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.949725 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:02:08.950000 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.949975 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" podUID="02382b30-a4fe-4eee-b624-8892f2d6e593" containerName="payload-processing" containerID="cri-o://d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35" gracePeriod=30 Sep 10 15:02:08.966197 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.966169 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:02:08.966546 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:08.966425 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" podUID="404fc626-a7d8-4986-97fe-8d9b42209121" containerName="payload-pre-processing" containerID="cri-o://d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea" gracePeriod=30 Sep 10 15:02:09.012259 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.012220 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:09.122693 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.122656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:09.217618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.217563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:09.218424 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.218397 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d"] Sep 10 15:02:09.220543 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.220516 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24e113dd-rp62d"] Sep 10 15:02:09.246181 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.246158 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:02:09.249286 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.249267 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:02:09.302516 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.302486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:09.374805 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.374768 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbm8n\" (UniqueName: \"kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n\") pod \"02382b30-a4fe-4eee-b624-8892f2d6e593\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " Sep 10 15:02:09.374984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.374843 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bnbk\" (UniqueName: \"kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk\") pod \"404fc626-a7d8-4986-97fe-8d9b42209121\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " Sep 10 15:02:09.374984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.374876 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume\") pod \"02382b30-a4fe-4eee-b624-8892f2d6e593\" (UID: \"02382b30-a4fe-4eee-b624-8892f2d6e593\") " Sep 10 15:02:09.374984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.374908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume\") pod \"404fc626-a7d8-4986-97fe-8d9b42209121\" (UID: \"404fc626-a7d8-4986-97fe-8d9b42209121\") " Sep 10 15:02:09.375285 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.375259 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02382b30-a4fe-4eee-b624-8892f2d6e593" (UID: "02382b30-a4fe-4eee-b624-8892f2d6e593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:09.375370 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.375341 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "404fc626-a7d8-4986-97fe-8d9b42209121" (UID: "404fc626-a7d8-4986-97fe-8d9b42209121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:09.377186 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.377158 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n" (OuterVolumeSpecName: "kube-api-access-rbm8n") pod "02382b30-a4fe-4eee-b624-8892f2d6e593" (UID: "02382b30-a4fe-4eee-b624-8892f2d6e593"). InnerVolumeSpecName "kube-api-access-rbm8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:09.377276 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.377239 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk" (OuterVolumeSpecName: "kube-api-access-5bnbk") pod "404fc626-a7d8-4986-97fe-8d9b42209121" (UID: "404fc626-a7d8-4986-97fe-8d9b42209121"). InnerVolumeSpecName "kube-api-access-5bnbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:09.393990 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.393950 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:09.463815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.463695 2581 generic.go:356] "Generic (PLEG): container finished" podID="404fc626-a7d8-4986-97fe-8d9b42209121" containerID="d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea" exitCode=0 Sep 10 15:02:09.463815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.463747 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" Sep 10 15:02:09.463815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.463802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" event={"ID":"404fc626-a7d8-4986-97fe-8d9b42209121","Type":"ContainerDied","Data":"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea"} Sep 10 15:02:09.464104 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.463857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr" event={"ID":"404fc626-a7d8-4986-97fe-8d9b42209121","Type":"ContainerDied","Data":"e323984e335eae7f3e900996ee0286988d6569098fdf31ab3aa53c5d190e82be"} Sep 10 15:02:09.464104 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.463882 2581 scope.go:117] "RemoveContainer" containerID="d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea" Sep 10 15:02:09.465291 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.465267 2581 generic.go:356] "Generic (PLEG): container finished" podID="02382b30-a4fe-4eee-b624-8892f2d6e593" containerID="d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35" exitCode=0 Sep 10 15:02:09.465383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.465315 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" Sep 10 15:02:09.465383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.465349 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" event={"ID":"02382b30-a4fe-4eee-b624-8892f2d6e593","Type":"ContainerDied","Data":"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35"} Sep 10 15:02:09.465494 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.465409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq" event={"ID":"02382b30-a4fe-4eee-b624-8892f2d6e593","Type":"ContainerDied","Data":"1d20dec66c796cdf531665f23750e987a3c25803cb3ea9a83ebf75b1a0533da9"} Sep 10 15:02:09.474909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.474884 2581 scope.go:117] "RemoveContainer" containerID="d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea" Sep 10 15:02:09.475251 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:02:09.475228 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea\": container with ID starting with d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea not found: ID does not exist" containerID="d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea" Sep 10 15:02:09.475342 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475267 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea"} err="failed to get container status \"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea\": rpc error: code = NotFound desc = could not find container \"d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea\": container with ID starting with d29f87cf1342217e46657474d3812bc9da59f5a946773befa359d950187303ea not found: ID does not exist" Sep 10 15:02:09.475342 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475288 2581 scope.go:117] "RemoveContainer" containerID="d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35" Sep 10 15:02:09.475684 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475660 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bnbk\" (UniqueName: \"kubernetes.io/projected/404fc626-a7d8-4986-97fe-8d9b42209121-kube-api-access-5bnbk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.475788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475691 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02382b30-a4fe-4eee-b624-8892f2d6e593-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.475788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475705 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404fc626-a7d8-4986-97fe-8d9b42209121-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.475788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.475718 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm8n\" (UniqueName: \"kubernetes.io/projected/02382b30-a4fe-4eee-b624-8892f2d6e593-kube-api-access-rbm8n\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.484648 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.484628 2581 scope.go:117] "RemoveContainer" containerID="d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35" Sep 10 15:02:09.485032 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:02:09.485013 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35\": container with ID starting with d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35 not found: ID does not exist" containerID="d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35" Sep 10 15:02:09.485094 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.485041 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35"} err="failed to get container status \"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35\": rpc error: code = NotFound desc = could not find container \"d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35\": container with ID starting with d5f16304578011aab212ca2a243a7f8fa2d061f88e75e0486295adf1a4a13d35 not found: ID does not exist" Sep 10 15:02:09.495323 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.495301 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:02:09.498950 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.498925 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-24e113dd-7d9965d86c-lv9nq"] Sep 10 15:02:09.511678 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.511656 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:02:09.519498 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.519476 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24e113dd-ddc975bfd-z2zzr"] Sep 10 15:02:09.908910 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:09.908870 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:10.018832 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.018726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:10.118006 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.117977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:10.230701 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.230670 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02382b30-a4fe-4eee-b624-8892f2d6e593" path="/var/lib/kubelet/pods/02382b30-a4fe-4eee-b624-8892f2d6e593/volumes" Sep 10 15:02:10.231061 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.231048 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404fc626-a7d8-4986-97fe-8d9b42209121" path="/var/lib/kubelet/pods/404fc626-a7d8-4986-97fe-8d9b42209121/volumes" Sep 10 15:02:10.231330 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.231311 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f8612b-b85a-456f-b13f-cc9f1447e0ef" path="/var/lib/kubelet/pods/94f8612b-b85a-456f-b13f-cc9f1447e0ef/volumes" Sep 10 15:02:10.450783 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.450652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:10.811685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:10.811649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:12.419215 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:12.419185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:02:12.419598 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:12.419255 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:02:13.311025 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:13.310980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:13.411423 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:13.411380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:13.592590 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:13.592495 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:13.694955 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:13.694919 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:13.971727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:13.971603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:14.105926 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.105889 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:14.198905 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.198863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:14.242116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242084 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:14.242499 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242483 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02382b30-a4fe-4eee-b624-8892f2d6e593" containerName="payload-processing" Sep 10 15:02:14.242499 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242497 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02382b30-a4fe-4eee-b624-8892f2d6e593" containerName="payload-processing" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242511 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f8612b-b85a-456f-b13f-cc9f1447e0ef" containerName="revoke-keys" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242518 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f8612b-b85a-456f-b13f-cc9f1447e0ef" containerName="revoke-keys" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242527 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404fc626-a7d8-4986-97fe-8d9b42209121" containerName="payload-pre-processing" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242534 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="404fc626-a7d8-4986-97fe-8d9b42209121" containerName="payload-pre-processing" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242607 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02382b30-a4fe-4eee-b624-8892f2d6e593" containerName="payload-processing" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242615 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f8612b-b85a-456f-b13f-cc9f1447e0ef" containerName="revoke-keys" Sep 10 15:02:14.242674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.242623 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="404fc626-a7d8-4986-97fe-8d9b42209121" containerName="payload-pre-processing" Sep 10 15:02:14.245579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.245560 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.253294 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.253261 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-12631b8d\"" Sep 10 15:02:14.253404 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.253386 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-12631b8d-dockercfg-j27jq\"" Sep 10 15:02:14.263625 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.263597 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:14.298474 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.298434 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:14.301054 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.301028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.315602 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.315569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptl4b\" (UniqueName: \"kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.315856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.315649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rppgt\" (UniqueName: \"kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.315856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.315686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.316383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.315849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.317362 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.317341 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:14.416378 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.416345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptl4b\" (UniqueName: \"kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.416603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.416405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rppgt\" (UniqueName: \"kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.416603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.416440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.416603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.416486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.417282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.417258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.417383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.417365 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.424998 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.424962 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:14.425622 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.425600 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rppgt\" (UniqueName: \"kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt\") pod \"payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.425713 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.425691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptl4b\" (UniqueName: \"kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b\") pod \"payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.556347 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.556257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:14.611838 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.611805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:14.842512 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.842486 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:14.845077 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:14.845050 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72a0adc_dab5_441a_80eb_4525562f6f06.slice/crio-4a944de02d80302e0bce4410f7e954d479a85c8e57b6de7008cc249aae50c421 WatchSource:0}: Error finding container 4a944de02d80302e0bce4410f7e954d479a85c8e57b6de7008cc249aae50c421: Status 404 returned error can't find the container with id 4a944de02d80302e0bce4410f7e954d479a85c8e57b6de7008cc249aae50c421 Sep 10 15:02:14.934822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:14.934794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:14.936924 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:14.936888 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6531bb3d_0cbd_4d4e_b295_c2c681109742.slice/crio-47b1a1da9008aae5453e75cf9cf9ed797232f4cfd1d002dd7cdd50fea0c27758 WatchSource:0}: Error finding container 47b1a1da9008aae5453e75cf9cf9ed797232f4cfd1d002dd7cdd50fea0c27758: Status 404 returned error can't find the container with id 47b1a1da9008aae5453e75cf9cf9ed797232f4cfd1d002dd7cdd50fea0c27758 Sep 10 15:02:15.505522 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.505460 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" event={"ID":"b72a0adc-dab5-441a-80eb-4525562f6f06","Type":"ContainerStarted","Data":"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055"} Sep 10 15:02:15.505750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.505541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:15.505750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.505567 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" event={"ID":"b72a0adc-dab5-441a-80eb-4525562f6f06","Type":"ContainerStarted","Data":"4a944de02d80302e0bce4410f7e954d479a85c8e57b6de7008cc249aae50c421"} Sep 10 15:02:15.507439 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.507413 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" event={"ID":"6531bb3d-0cbd-4d4e-b295-c2c681109742","Type":"ContainerStarted","Data":"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50"} Sep 10 15:02:15.507561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.507461 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" event={"ID":"6531bb3d-0cbd-4d4e-b295-c2c681109742","Type":"ContainerStarted","Data":"47b1a1da9008aae5453e75cf9cf9ed797232f4cfd1d002dd7cdd50fea0c27758"} Sep 10 15:02:15.507561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.507509 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:15.525129 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.525078 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" podStartSLOduration=1.5250613400000002 podStartE2EDuration="1.52506134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:15.523439399 +0000 UTC m=+889.817741531" watchObservedRunningTime="2026-09-10 15:02:15.52506134 +0000 UTC m=+889.819363502" Sep 10 15:02:15.539423 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:15.539360 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" podStartSLOduration=1.539339371 podStartE2EDuration="1.539339371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:15.53897455 +0000 UTC m=+889.833276680" watchObservedRunningTime="2026-09-10 15:02:15.539339371 +0000 UTC m=+889.833641502" Sep 10 15:02:20.398303 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:20.398215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:22.655957 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:22.655923 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:26.191109 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:26.191079 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:02:26.191833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:26.191815 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:02:26.515582 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:26.515544 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:26.515777 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:26.515615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:27.964956 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:27.964922 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:34.728418 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:34.728380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:38.259252 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.259218 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b"] Sep 10 15:02:38.262018 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.261994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:38.264283 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.264264 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:02:38.271123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.271095 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b"] Sep 10 15:02:38.344995 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.344959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b\" (UID: \"cfc2590a-9e84-40e4-bc70-149e08650a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:38.445696 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.445656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b\" (UID: \"cfc2590a-9e84-40e4-bc70-149e08650a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:38.446409 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.446383 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b\" (UID: \"cfc2590a-9e84-40e4-bc70-149e08650a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:38.575036 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.574947 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:38.851596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:38.851565 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b"] Sep 10 15:02:38.852960 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:38.852930 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc2590a_9e84_40e4_bc70_149e08650a1e.slice/crio-a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a WatchSource:0}: Error finding container a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a: Status 404 returned error can't find the container with id a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a Sep 10 15:02:39.649134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:39.649102 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc2590a-9e84-40e4-bc70-149e08650a1e" containerID="c98502372b8375baea722a742ee511b8bc45df0a55c69ca8c3208eb97f1657dd" exitCode=0 Sep 10 15:02:39.649517 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:39.649148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" event={"ID":"cfc2590a-9e84-40e4-bc70-149e08650a1e","Type":"ContainerDied","Data":"c98502372b8375baea722a742ee511b8bc45df0a55c69ca8c3208eb97f1657dd"} Sep 10 15:02:39.649517 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:39.649196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" event={"ID":"cfc2590a-9e84-40e4-bc70-149e08650a1e","Type":"ContainerStarted","Data":"a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a"} Sep 10 15:02:40.809490 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:40.809463 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:40.867542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:40.867500 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca\") pod \"cfc2590a-9e84-40e4-bc70-149e08650a1e\" (UID: \"cfc2590a-9e84-40e4-bc70-149e08650a1e\") " Sep 10 15:02:40.867853 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:40.867828 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfc2590a-9e84-40e4-bc70-149e08650a1e" (UID: "cfc2590a-9e84-40e4-bc70-149e08650a1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:40.968906 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:40.968811 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc2590a-9e84-40e4-bc70-149e08650a1e-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:41.660831 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:41.660799 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" Sep 10 15:02:41.661006 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:41.660792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b" event={"ID":"cfc2590a-9e84-40e4-bc70-149e08650a1e","Type":"ContainerDied","Data":"a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a"} Sep 10 15:02:41.661006 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:41.660918 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61836cdfd6a4a92cc311a07611895e8ec178775755610f18f17f32d4008203a" Sep 10 15:02:44.190487 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:44.190450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.125192 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.125161 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.159456 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.159428 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:45.159674 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.159649 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" podUID="b72a0adc-dab5-441a-80eb-4525562f6f06" containerName="payload-processing" containerID="cri-o://07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055" gracePeriod=30 Sep 10 15:02:45.165396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.165370 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:45.165750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.165723 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" podUID="6531bb3d-0cbd-4d4e-b295-c2c681109742" containerName="payload-pre-processing" containerID="cri-o://3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50" gracePeriod=30 Sep 10 15:02:45.215820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.215783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.296276 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.296218 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.387483 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.387443 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b"] Sep 10 15:02:45.393793 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.392902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.393962 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.393849 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-12631b8d-mc28b"] Sep 10 15:02:45.445936 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.445896 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:45.463982 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.463957 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:45.472836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.472803 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.520172 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.520140 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume\") pod \"b72a0adc-dab5-441a-80eb-4525562f6f06\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " Sep 10 15:02:45.520362 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.520182 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptl4b\" (UniqueName: \"kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b\") pod \"b72a0adc-dab5-441a-80eb-4525562f6f06\" (UID: \"b72a0adc-dab5-441a-80eb-4525562f6f06\") " Sep 10 15:02:45.520627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.520594 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b72a0adc-dab5-441a-80eb-4525562f6f06" (UID: "b72a0adc-dab5-441a-80eb-4525562f6f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:45.523140 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.522697 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b" (OuterVolumeSpecName: "kube-api-access-ptl4b") pod "b72a0adc-dab5-441a-80eb-4525562f6f06" (UID: "b72a0adc-dab5-441a-80eb-4525562f6f06"). InnerVolumeSpecName "kube-api-access-ptl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:45.621490 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.621447 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume\") pod \"6531bb3d-0cbd-4d4e-b295-c2c681109742\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " Sep 10 15:02:45.621667 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.621583 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rppgt\" (UniqueName: \"kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt\") pod \"6531bb3d-0cbd-4d4e-b295-c2c681109742\" (UID: \"6531bb3d-0cbd-4d4e-b295-c2c681109742\") " Sep 10 15:02:45.621889 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.621869 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72a0adc-dab5-441a-80eb-4525562f6f06-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:45.621935 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.621896 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptl4b\" (UniqueName: \"kubernetes.io/projected/b72a0adc-dab5-441a-80eb-4525562f6f06-kube-api-access-ptl4b\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:45.621970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.621943 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6531bb3d-0cbd-4d4e-b295-c2c681109742" (UID: "6531bb3d-0cbd-4d4e-b295-c2c681109742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:45.623844 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.623818 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt" (OuterVolumeSpecName: "kube-api-access-rppgt") pod "6531bb3d-0cbd-4d4e-b295-c2c681109742" (UID: "6531bb3d-0cbd-4d4e-b295-c2c681109742"). InnerVolumeSpecName "kube-api-access-rppgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:45.668471 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.668375 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:45.683423 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.683387 2581 generic.go:356] "Generic (PLEG): container finished" podID="b72a0adc-dab5-441a-80eb-4525562f6f06" containerID="07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055" exitCode=0 Sep 10 15:02:45.683601 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.683435 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" Sep 10 15:02:45.683601 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.683476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" event={"ID":"b72a0adc-dab5-441a-80eb-4525562f6f06","Type":"ContainerDied","Data":"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055"} Sep 10 15:02:45.683601 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.683526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b" event={"ID":"b72a0adc-dab5-441a-80eb-4525562f6f06","Type":"ContainerDied","Data":"4a944de02d80302e0bce4410f7e954d479a85c8e57b6de7008cc249aae50c421"} Sep 10 15:02:45.683601 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.683549 2581 scope.go:117] "RemoveContainer" containerID="07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055" Sep 10 15:02:45.685095 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.685073 2581 generic.go:356] "Generic (PLEG): container finished" podID="6531bb3d-0cbd-4d4e-b295-c2c681109742" containerID="3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50" exitCode=0 Sep 10 15:02:45.685181 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.685154 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" event={"ID":"6531bb3d-0cbd-4d4e-b295-c2c681109742","Type":"ContainerDied","Data":"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50"} Sep 10 15:02:45.685240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.685188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" event={"ID":"6531bb3d-0cbd-4d4e-b295-c2c681109742","Type":"ContainerDied","Data":"47b1a1da9008aae5453e75cf9cf9ed797232f4cfd1d002dd7cdd50fea0c27758"} Sep 10 15:02:45.685285 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.685250 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4" Sep 10 15:02:45.694667 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.694622 2581 scope.go:117] "RemoveContainer" containerID="07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055" Sep 10 15:02:45.695073 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:02:45.695046 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055\": container with ID starting with 07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055 not found: ID does not exist" containerID="07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055" Sep 10 15:02:45.695153 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.695086 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055"} err="failed to get container status \"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055\": rpc error: code = NotFound desc = could not find container \"07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055\": container with ID starting with 07c5eeebc6e5532fdd5c6d61488abda8d101f4b36f7b6d1aa4455948b6874055 not found: ID does not exist" Sep 10 15:02:45.695153 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.695109 2581 scope.go:117] "RemoveContainer" containerID="3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50" Sep 10 15:02:45.704586 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.704568 2581 scope.go:117] "RemoveContainer" containerID="3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50" Sep 10 15:02:45.704926 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:02:45.704905 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50\": container with ID starting with 3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50 not found: ID does not exist" containerID="3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50" Sep 10 15:02:45.704993 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.704934 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50"} err="failed to get container status \"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50\": rpc error: code = NotFound desc = could not find container \"3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50\": container with ID starting with 3e5fee273ea027a8af44ffe8f1b123be95c4e5bb03359f1eb6ff69423672da50 not found: ID does not exist" Sep 10 15:02:45.715801 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.715778 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:45.722435 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.722405 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rppgt\" (UniqueName: \"kubernetes.io/projected/6531bb3d-0cbd-4d4e-b295-c2c681109742-kube-api-access-rppgt\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:45.722435 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.722435 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6531bb3d-0cbd-4d4e-b295-c2c681109742-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:02:45.724310 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.724289 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-12631b8d-788857846b-fqs5b"] Sep 10 15:02:45.735130 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.735104 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:45.751691 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:45.751662 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-12631b8d-6dd8bb8778-s8wp4"] Sep 10 15:02:46.026078 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.026026 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:46.232123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.232081 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6531bb3d-0cbd-4d4e-b295-c2c681109742" path="/var/lib/kubelet/pods/6531bb3d-0cbd-4d4e-b295-c2c681109742/volumes" Sep 10 15:02:46.232497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.232437 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72a0adc-dab5-441a-80eb-4525562f6f06" path="/var/lib/kubelet/pods/b72a0adc-dab5-441a-80eb-4525562f6f06/volumes" Sep 10 15:02:46.232711 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.232697 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc2590a-9e84-40e4-bc70-149e08650a1e" path="/var/lib/kubelet/pods/cfc2590a-9e84-40e4-bc70-149e08650a1e/volumes" Sep 10 15:02:46.348737 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.348641 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:46.529531 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:46.529498 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:48.363580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:48.363548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:48.459612 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:48.459579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:49.524633 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:49.524593 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:49.624804 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:49.624747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:49.729396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:49.729339 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:50.196580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.196547 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:50.295629 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.295576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:02:50.306251 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306214 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:02:50.306644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306624 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6531bb3d-0cbd-4d4e-b295-c2c681109742" containerName="payload-pre-processing" Sep 10 15:02:50.306644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306645 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6531bb3d-0cbd-4d4e-b295-c2c681109742" containerName="payload-pre-processing" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306658 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc2590a-9e84-40e4-bc70-149e08650a1e" containerName="revoke-keys" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306664 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc2590a-9e84-40e4-bc70-149e08650a1e" containerName="revoke-keys" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306689 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72a0adc-dab5-441a-80eb-4525562f6f06" containerName="payload-processing" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306695 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72a0adc-dab5-441a-80eb-4525562f6f06" containerName="payload-processing" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306776 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6531bb3d-0cbd-4d4e-b295-c2c681109742" containerName="payload-pre-processing" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306789 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc2590a-9e84-40e4-bc70-149e08650a1e" containerName="revoke-keys" Sep 10 15:02:50.306833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.306798 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72a0adc-dab5-441a-80eb-4525562f6f06" containerName="payload-processing" Sep 10 15:02:50.309845 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.309826 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.312177 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.312158 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-78cfd1dc-dockercfg-9z9r5\"" Sep 10 15:02:50.312290 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.312165 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-78cfd1dc\"" Sep 10 15:02:50.322467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.322436 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:02:50.338235 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.338194 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:02:50.341411 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.341386 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.353060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.353035 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:02:50.464986 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.464887 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfln\" (UniqueName: \"kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.464986 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.464961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.465199 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.465019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zts6\" (UniqueName: \"kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.465199 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.465069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.566380 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.566332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.566865 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.566400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfln\" (UniqueName: \"kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.566865 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.566440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.566865 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.566484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zts6\" (UniqueName: \"kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.567222 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.567202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.567260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.567202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.574607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.574576 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zts6\" (UniqueName: \"kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6\") pod \"payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.574844 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.574822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfln\" (UniqueName: \"kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln\") pod \"payload-processing-e2e-ait-78cfd1dc-667676547f-57p89\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.620505 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.620466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:50.652692 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.652632 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:50.900054 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.900025 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:02:50.902452 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:50.902422 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5165350b_aafb_46e4_8085_7ef427a35aa6.slice/crio-40bb4401f3471efe673ca9e0e9964b9815476255d9cc30d64764b1bb5a03ec04 WatchSource:0}: Error finding container 40bb4401f3471efe673ca9e0e9964b9815476255d9cc30d64764b1bb5a03ec04: Status 404 returned error can't find the container with id 40bb4401f3471efe673ca9e0e9964b9815476255d9cc30d64764b1bb5a03ec04 Sep 10 15:02:50.949574 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:50.949540 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:02:50.954105 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:02:50.954073 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d95b195_966d_4f6d_bd30_973fb2a945f9.slice/crio-3c4bb8ac4ef4bb8786bdd30ffdb44ac11332389a42d40a0caec742fe472ac1e1 WatchSource:0}: Error finding container 3c4bb8ac4ef4bb8786bdd30ffdb44ac11332389a42d40a0caec742fe472ac1e1: Status 404 returned error can't find the container with id 3c4bb8ac4ef4bb8786bdd30ffdb44ac11332389a42d40a0caec742fe472ac1e1 Sep 10 15:02:51.726071 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.726018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" event={"ID":"8d95b195-966d-4f6d-bd30-973fb2a945f9","Type":"ContainerStarted","Data":"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267"} Sep 10 15:02:51.726580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.726089 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:02:51.726580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.726115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" event={"ID":"8d95b195-966d-4f6d-bd30-973fb2a945f9","Type":"ContainerStarted","Data":"3c4bb8ac4ef4bb8786bdd30ffdb44ac11332389a42d40a0caec742fe472ac1e1"} Sep 10 15:02:51.728127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.728094 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" event={"ID":"5165350b-aafb-46e4-8085-7ef427a35aa6","Type":"ContainerStarted","Data":"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095"} Sep 10 15:02:51.728271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.728162 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:02:51.728271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.728189 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" event={"ID":"5165350b-aafb-46e4-8085-7ef427a35aa6","Type":"ContainerStarted","Data":"40bb4401f3471efe673ca9e0e9964b9815476255d9cc30d64764b1bb5a03ec04"} Sep 10 15:02:51.745830 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.745747 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" podStartSLOduration=1.7457291339999998 podStartE2EDuration="1.745729134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:51.743645477 +0000 UTC m=+926.037947607" watchObservedRunningTime="2026-09-10 15:02:51.745729134 +0000 UTC m=+926.040031282" Sep 10 15:02:51.761431 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:02:51.761376 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" podStartSLOduration=1.7613576210000002 podStartE2EDuration="1.761357621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:51.760044986 +0000 UTC m=+926.054347118" watchObservedRunningTime="2026-09-10 15:02:51.761357621 +0000 UTC m=+926.055659793" Sep 10 15:03:02.735477 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:02.735438 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:03:02.735974 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:02.735666 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:03:07.546049 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:07.545975 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:12.151107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:12.151063 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:14.033157 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.033132 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t"] Sep 10 15:03:14.036234 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.036216 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:14.038565 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.038539 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:03:14.043921 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.043896 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t"] Sep 10 15:03:14.069221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.069187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t\" (UID: \"712e3a80-49fb-49b6-80ac-1c6ac3472118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:14.170344 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.170305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t\" (UID: \"712e3a80-49fb-49b6-80ac-1c6ac3472118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:14.170993 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.170969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t\" (UID: \"712e3a80-49fb-49b6-80ac-1c6ac3472118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:14.197035 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.196993 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:14.352706 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.352622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:14.629927 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.629858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t"] Sep 10 15:03:14.632320 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:03:14.632287 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712e3a80_49fb_49b6_80ac_1c6ac3472118.slice/crio-073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4 WatchSource:0}: Error finding container 073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4: Status 404 returned error can't find the container with id 073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4 Sep 10 15:03:14.875520 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:14.875478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" event={"ID":"712e3a80-49fb-49b6-80ac-1c6ac3472118","Type":"ContainerStarted","Data":"073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4"} Sep 10 15:03:15.883037 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:15.882997 2581 generic.go:356] "Generic (PLEG): container finished" podID="712e3a80-49fb-49b6-80ac-1c6ac3472118" containerID="efe220fbb02492f4a86f12557ca5581e946487257017c9a37bd458a36126367d" exitCode=0 Sep 10 15:03:15.883454 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:15.883076 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" event={"ID":"712e3a80-49fb-49b6-80ac-1c6ac3472118","Type":"ContainerDied","Data":"efe220fbb02492f4a86f12557ca5581e946487257017c9a37bd458a36126367d"} Sep 10 15:03:17.033127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.033100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:17.098913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.098874 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca\") pod \"712e3a80-49fb-49b6-80ac-1c6ac3472118\" (UID: \"712e3a80-49fb-49b6-80ac-1c6ac3472118\") " Sep 10 15:03:17.099304 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.099275 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "712e3a80-49fb-49b6-80ac-1c6ac3472118" (UID: "712e3a80-49fb-49b6-80ac-1c6ac3472118"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:17.200167 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.200077 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e3a80-49fb-49b6-80ac-1c6ac3472118-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:17.325649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.325609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx"] Sep 10 15:03:17.326122 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.326108 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712e3a80-49fb-49b6-80ac-1c6ac3472118" containerName="revoke-keys" Sep 10 15:03:17.326178 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.326124 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="712e3a80-49fb-49b6-80ac-1c6ac3472118" containerName="revoke-keys" Sep 10 15:03:17.326228 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.326218 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="712e3a80-49fb-49b6-80ac-1c6ac3472118" containerName="revoke-keys" Sep 10 15:03:17.328383 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.328365 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:17.336308 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.336273 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx"] Sep 10 15:03:17.401634 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.401594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c75e68-lv7qx\" (UID: \"e3198c99-d510-4e1e-bd02-db19e14a4884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:17.481871 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.481785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:17.503041 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.503002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c75e68-lv7qx\" (UID: \"e3198c99-d510-4e1e-bd02-db19e14a4884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:17.503687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.503662 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c75e68-lv7qx\" (UID: \"e3198c99-d510-4e1e-bd02-db19e14a4884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:17.641358 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.641317 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:17.898311 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.898279 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" Sep 10 15:03:17.898501 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.898329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t" event={"ID":"712e3a80-49fb-49b6-80ac-1c6ac3472118","Type":"ContainerDied","Data":"073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4"} Sep 10 15:03:17.898501 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.898386 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="073d893f000d380f04f6f8bbd72b8f2e810b258862967035fbdd64582a21a9e4" Sep 10 15:03:17.917098 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:17.916943 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx"] Sep 10 15:03:17.919504 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:03:17.919479 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3198c99_d510_4e1e_bd02_db19e14a4884.slice/crio-e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3 WatchSource:0}: Error finding container e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3: Status 404 returned error can't find the container with id e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3 Sep 10 15:03:18.906189 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:18.906155 2581 generic.go:356] "Generic (PLEG): container finished" podID="e3198c99-d510-4e1e-bd02-db19e14a4884" containerID="a276a396a5dbcc0d1530fd00bd1d6cf120ddcd7dd48380cda318a50c8a6e845f" exitCode=0 Sep 10 15:03:18.906622 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:18.906250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" event={"ID":"e3198c99-d510-4e1e-bd02-db19e14a4884","Type":"ContainerDied","Data":"a276a396a5dbcc0d1530fd00bd1d6cf120ddcd7dd48380cda318a50c8a6e845f"} Sep 10 15:03:18.906622 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:18.906329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" event={"ID":"e3198c99-d510-4e1e-bd02-db19e14a4884","Type":"ContainerStarted","Data":"e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3"} Sep 10 15:03:19.795972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:19.795933 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.056531 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.056466 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:20.127379 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.127344 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca\") pod \"e3198c99-d510-4e1e-bd02-db19e14a4884\" (UID: \"e3198c99-d510-4e1e-bd02-db19e14a4884\") " Sep 10 15:03:20.127664 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.127641 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3198c99-d510-4e1e-bd02-db19e14a4884" (UID: "e3198c99-d510-4e1e-bd02-db19e14a4884"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:20.228449 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.228420 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3198c99-d510-4e1e-bd02-db19e14a4884-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.400919 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.400830 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.434668 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.434611 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:03:20.435220 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.435186 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" podUID="8d95b195-966d-4f6d-bd30-973fb2a945f9" containerName="payload-processing" containerID="cri-o://ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267" gracePeriod=30 Sep 10 15:03:20.439739 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.439708 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:03:20.440450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.439961 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" podUID="5165350b-aafb-46e4-8085-7ef427a35aa6" containerName="payload-pre-processing" containerID="cri-o://17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095" gracePeriod=30 Sep 10 15:03:20.475615 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.475572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.536177 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.536144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.608369 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.608336 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t"] Sep 10 15:03:20.614946 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.614908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.616275 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.616242 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78cfd1dc-55x9t"] Sep 10 15:03:20.680774 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.680720 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:20.707712 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.707689 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:03:20.725324 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.725289 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:03:20.834266 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834226 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume\") pod \"5165350b-aafb-46e4-8085-7ef427a35aa6\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " Sep 10 15:03:20.834467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834358 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zts6\" (UniqueName: \"kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6\") pod \"5165350b-aafb-46e4-8085-7ef427a35aa6\" (UID: \"5165350b-aafb-46e4-8085-7ef427a35aa6\") " Sep 10 15:03:20.834467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834379 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume\") pod \"8d95b195-966d-4f6d-bd30-973fb2a945f9\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " Sep 10 15:03:20.834467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834403 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tfln\" (UniqueName: \"kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln\") pod \"8d95b195-966d-4f6d-bd30-973fb2a945f9\" (UID: \"8d95b195-966d-4f6d-bd30-973fb2a945f9\") " Sep 10 15:03:20.834636 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834615 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5165350b-aafb-46e4-8085-7ef427a35aa6" (UID: "5165350b-aafb-46e4-8085-7ef427a35aa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:20.834748 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.834723 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d95b195-966d-4f6d-bd30-973fb2a945f9" (UID: "8d95b195-966d-4f6d-bd30-973fb2a945f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:20.836836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.836811 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6" (OuterVolumeSpecName: "kube-api-access-8zts6") pod "5165350b-aafb-46e4-8085-7ef427a35aa6" (UID: "5165350b-aafb-46e4-8085-7ef427a35aa6"). InnerVolumeSpecName "kube-api-access-8zts6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:20.836836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.836821 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln" (OuterVolumeSpecName: "kube-api-access-5tfln") pod "8d95b195-966d-4f6d-bd30-973fb2a945f9" (UID: "8d95b195-966d-4f6d-bd30-973fb2a945f9"). InnerVolumeSpecName "kube-api-access-5tfln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:20.919520 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.919431 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d95b195-966d-4f6d-bd30-973fb2a945f9" containerID="ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267" exitCode=0 Sep 10 15:03:20.919520 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.919472 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" Sep 10 15:03:20.919729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.919514 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" event={"ID":"8d95b195-966d-4f6d-bd30-973fb2a945f9","Type":"ContainerDied","Data":"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267"} Sep 10 15:03:20.919729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.919563 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89" event={"ID":"8d95b195-966d-4f6d-bd30-973fb2a945f9","Type":"ContainerDied","Data":"3c4bb8ac4ef4bb8786bdd30ffdb44ac11332389a42d40a0caec742fe472ac1e1"} Sep 10 15:03:20.919729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.919585 2581 scope.go:117] "RemoveContainer" containerID="ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267" Sep 10 15:03:20.921025 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.921005 2581 generic.go:356] "Generic (PLEG): container finished" podID="5165350b-aafb-46e4-8085-7ef427a35aa6" containerID="17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095" exitCode=0 Sep 10 15:03:20.921113 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.921048 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" Sep 10 15:03:20.921113 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.921075 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" event={"ID":"5165350b-aafb-46e4-8085-7ef427a35aa6","Type":"ContainerDied","Data":"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095"} Sep 10 15:03:20.921218 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.921121 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr" event={"ID":"5165350b-aafb-46e4-8085-7ef427a35aa6","Type":"ContainerDied","Data":"40bb4401f3471efe673ca9e0e9964b9815476255d9cc30d64764b1bb5a03ec04"} Sep 10 15:03:20.922707 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.922683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" event={"ID":"e3198c99-d510-4e1e-bd02-db19e14a4884","Type":"ContainerDied","Data":"e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3"} Sep 10 15:03:20.922707 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.922702 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx" Sep 10 15:03:20.922876 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.922729 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5021bd9ea2d279791f021fc1b350758cbe922b5c0a9ddb5791b645a4c44a8c3" Sep 10 15:03:20.933063 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.933036 2581 scope.go:117] "RemoveContainer" containerID="ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267" Sep 10 15:03:20.933501 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:03:20.933475 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267\": container with ID starting with ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267 not found: ID does not exist" containerID="ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267" Sep 10 15:03:20.933616 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.933514 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267"} err="failed to get container status \"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267\": rpc error: code = NotFound desc = could not find container \"ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267\": container with ID starting with ff3f282c08b368ca9daab6ead8759c455eef50ff19b3f87cb5d2403c46c11267 not found: ID does not exist" Sep 10 15:03:20.933616 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.933536 2581 scope.go:117] "RemoveContainer" containerID="17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095" Sep 10 15:03:20.935966 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.935948 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zts6\" (UniqueName: \"kubernetes.io/projected/5165350b-aafb-46e4-8085-7ef427a35aa6-kube-api-access-8zts6\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.936033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.935968 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d95b195-966d-4f6d-bd30-973fb2a945f9-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.936033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.935978 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfln\" (UniqueName: \"kubernetes.io/projected/8d95b195-966d-4f6d-bd30-973fb2a945f9-kube-api-access-5tfln\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.936033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.935988 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5165350b-aafb-46e4-8085-7ef427a35aa6-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.942876 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.942857 2581 scope.go:117] "RemoveContainer" containerID="17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095" Sep 10 15:03:20.943203 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:03:20.943183 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095\": container with ID starting with 17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095 not found: ID does not exist" containerID="17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095" Sep 10 15:03:20.943281 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.943210 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095"} err="failed to get container status \"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095\": rpc error: code = NotFound desc = could not find container \"17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095\": container with ID starting with 17e626f5b0bbf3cd99b0b375d71b086e44cf28e75045f4904bc026dd39ace095 not found: ID does not exist" Sep 10 15:03:20.946625 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.946602 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:03:20.951201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.951177 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78cfd1dc-667676547f-57p89"] Sep 10 15:03:20.962935 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.962904 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:03:20.967727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:20.967703 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78cfd1dc-694cdfccf8-2dhkr"] Sep 10 15:03:21.700940 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:21.700908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:22.120944 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.120906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:22.232484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.232448 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5165350b-aafb-46e4-8085-7ef427a35aa6" path="/var/lib/kubelet/pods/5165350b-aafb-46e4-8085-7ef427a35aa6/volumes" Sep 10 15:03:22.232915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.232893 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712e3a80-49fb-49b6-80ac-1c6ac3472118" path="/var/lib/kubelet/pods/712e3a80-49fb-49b6-80ac-1c6ac3472118/volumes" Sep 10 15:03:22.233297 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.233280 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d95b195-966d-4f6d-bd30-973fb2a945f9" path="/var/lib/kubelet/pods/8d95b195-966d-4f6d-bd30-973fb2a945f9/volumes" Sep 10 15:03:22.436283 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.436058 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:22.611000 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:22.610957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:24.563389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:24.563347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:24.664351 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:24.664323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:25.757004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:25.756965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:25.866666 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:25.866620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:25.957988 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:25.957868 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:26.371152 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.371117 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:26.462648 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.462603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:26.480158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480121 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:26.480588 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480571 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d95b195-966d-4f6d-bd30-973fb2a945f9" containerName="payload-processing" Sep 10 15:03:26.480647 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480593 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d95b195-966d-4f6d-bd30-973fb2a945f9" containerName="payload-processing" Sep 10 15:03:26.480647 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480626 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5165350b-aafb-46e4-8085-7ef427a35aa6" containerName="payload-pre-processing" Sep 10 15:03:26.480647 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480633 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5165350b-aafb-46e4-8085-7ef427a35aa6" containerName="payload-pre-processing" Sep 10 15:03:26.480745 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480652 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3198c99-d510-4e1e-bd02-db19e14a4884" containerName="revoke-keys" Sep 10 15:03:26.480745 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480658 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3198c99-d510-4e1e-bd02-db19e14a4884" containerName="revoke-keys" Sep 10 15:03:26.480841 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480748 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5165350b-aafb-46e4-8085-7ef427a35aa6" containerName="payload-pre-processing" Sep 10 15:03:26.480841 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480771 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d95b195-966d-4f6d-bd30-973fb2a945f9" containerName="payload-processing" Sep 10 15:03:26.480841 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.480778 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3198c99-d510-4e1e-bd02-db19e14a4884" containerName="revoke-keys" Sep 10 15:03:26.483922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.483901 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.486723 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.486700 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c3f76b1a\"" Sep 10 15:03:26.486723 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.486730 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c3f76b1a-dockercfg-4qrw2\"" Sep 10 15:03:26.500125 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.500094 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:26.502968 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.502944 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:26.503153 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.503034 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.520138 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.519400 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:26.592249 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.592146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.592249 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.592241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mj4v\" (UniqueName: \"kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.592453 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.592285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gcp\" (UniqueName: \"kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.592453 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.592316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.693730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.693680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.694028 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.693803 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mj4v\" (UniqueName: \"kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.694028 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.693847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gcp\" (UniqueName: \"kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.694028 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.693878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.694635 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.694605 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.694796 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.694695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.704261 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.704236 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mj4v\" (UniqueName: \"kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v\") pod \"payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.704739 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.704716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gcp\" (UniqueName: \"kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp\") pod \"payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:26.794801 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.794747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:26.814227 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:26.814195 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:27.078795 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.078767 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:27.080894 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:03:27.080866 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d984c75_181a_4816_8162_f1d2eb091a41.slice/crio-a15a1b8b79916911edb43d7c072dd1c20bafc2f0b8a0f0c9583fdadee31ae9b5 WatchSource:0}: Error finding container a15a1b8b79916911edb43d7c072dd1c20bafc2f0b8a0f0c9583fdadee31ae9b5: Status 404 returned error can't find the container with id a15a1b8b79916911edb43d7c072dd1c20bafc2f0b8a0f0c9583fdadee31ae9b5 Sep 10 15:03:27.101015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.100982 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:27.102954 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:03:27.102912 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44545ed_914e_448f_bf9c_19c19209d9de.slice/crio-4f049351c12314e396ac2767151efca787fa18e36ff11921809208b7f1f3fef9 WatchSource:0}: Error finding container 4f049351c12314e396ac2767151efca787fa18e36ff11921809208b7f1f3fef9: Status 404 returned error can't find the container with id 4f049351c12314e396ac2767151efca787fa18e36ff11921809208b7f1f3fef9 Sep 10 15:03:27.700005 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.699966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:27.729387 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.729342 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:03:27.729617 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.729592 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" podUID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" containerName="payload-processing" containerID="cri-o://f8115fe6930d68ad2e2b2e8d4936b11b48625c6b3ed5c6befda179b43e442ca0" gracePeriod=30 Sep 10 15:03:27.739100 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.739072 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:03:27.739330 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.739295 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" podUID="4683ba2b-3433-475c-a17f-2a31930df65a" containerName="payload-pre-processing" containerID="cri-o://f6cba88a6ed6973a38b6ecf91e9e516e512e2e804c7a3fc918849848099c19af" gracePeriod=30 Sep 10 15:03:27.775486 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.775429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:27.848184 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.848150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:27.924747 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.924198 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx"] Sep 10 15:03:27.927789 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.927477 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:27.931266 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.931241 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c75e68-lv7qx"] Sep 10 15:03:27.975189 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.975089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" event={"ID":"1d984c75-181a-4816-8162-f1d2eb091a41","Type":"ContainerStarted","Data":"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957"} Sep 10 15:03:27.975434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.975405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:27.975554 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.975486 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" event={"ID":"1d984c75-181a-4816-8162-f1d2eb091a41","Type":"ContainerStarted","Data":"a15a1b8b79916911edb43d7c072dd1c20bafc2f0b8a0f0c9583fdadee31ae9b5"} Sep 10 15:03:27.977605 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.977581 2581 generic.go:356] "Generic (PLEG): container finished" podID="4683ba2b-3433-475c-a17f-2a31930df65a" containerID="f6cba88a6ed6973a38b6ecf91e9e516e512e2e804c7a3fc918849848099c19af" exitCode=0 Sep 10 15:03:27.977716 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.977641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" event={"ID":"4683ba2b-3433-475c-a17f-2a31930df65a","Type":"ContainerDied","Data":"f6cba88a6ed6973a38b6ecf91e9e516e512e2e804c7a3fc918849848099c19af"} Sep 10 15:03:27.979597 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.979574 2581 generic.go:356] "Generic (PLEG): container finished" podID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" containerID="f8115fe6930d68ad2e2b2e8d4936b11b48625c6b3ed5c6befda179b43e442ca0" exitCode=0 Sep 10 15:03:27.979708 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.979648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" event={"ID":"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5","Type":"ContainerDied","Data":"f8115fe6930d68ad2e2b2e8d4936b11b48625c6b3ed5c6befda179b43e442ca0"} Sep 10 15:03:27.981799 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.981774 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" event={"ID":"f44545ed-914e-448f-bf9c-19c19209d9de","Type":"ContainerStarted","Data":"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d"} Sep 10 15:03:27.981882 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.981825 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" event={"ID":"f44545ed-914e-448f-bf9c-19c19209d9de","Type":"ContainerStarted","Data":"4f049351c12314e396ac2767151efca787fa18e36ff11921809208b7f1f3fef9"} Sep 10 15:03:27.982130 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.982112 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:27.995413 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.994743 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" podStartSLOduration=1.994721612 podStartE2EDuration="1.994721612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:27.992429759 +0000 UTC m=+962.286731889" watchObservedRunningTime="2026-09-10 15:03:27.994721612 +0000 UTC m=+962.289023743" Sep 10 15:03:27.996985 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:27.996790 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:28.011529 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.011447 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" podStartSLOduration=2.011431288 podStartE2EDuration="2.011431288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:28.010674211 +0000 UTC m=+962.304976342" watchObservedRunningTime="2026-09-10 15:03:28.011431288 +0000 UTC m=+962.305733417" Sep 10 15:03:28.026646 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.026616 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:03:28.051717 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.051692 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:03:28.108971 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.108931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k62cq\" (UniqueName: \"kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq\") pod \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " Sep 10 15:03:28.109165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109020 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume\") pod \"4683ba2b-3433-475c-a17f-2a31930df65a\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " Sep 10 15:03:28.109165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109042 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume\") pod \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\" (UID: \"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5\") " Sep 10 15:03:28.109165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109135 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4pcs\" (UniqueName: \"kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs\") pod \"4683ba2b-3433-475c-a17f-2a31930df65a\" (UID: \"4683ba2b-3433-475c-a17f-2a31930df65a\") " Sep 10 15:03:28.109382 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109362 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" (UID: "c30ffa16-cf70-49fe-8c3f-930e95a4f0c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:28.109450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109424 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4683ba2b-3433-475c-a17f-2a31930df65a" (UID: "4683ba2b-3433-475c-a17f-2a31930df65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:28.109579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109560 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683ba2b-3433-475c-a17f-2a31930df65a-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:28.109579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.109583 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:28.111990 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.111965 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq" (OuterVolumeSpecName: "kube-api-access-k62cq") pod "c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" (UID: "c30ffa16-cf70-49fe-8c3f-930e95a4f0c5"). InnerVolumeSpecName "kube-api-access-k62cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:28.111990 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.111978 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs" (OuterVolumeSpecName: "kube-api-access-v4pcs") pod "4683ba2b-3433-475c-a17f-2a31930df65a" (UID: "4683ba2b-3433-475c-a17f-2a31930df65a"). InnerVolumeSpecName "kube-api-access-v4pcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:28.210334 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.210295 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pcs\" (UniqueName: \"kubernetes.io/projected/4683ba2b-3433-475c-a17f-2a31930df65a-kube-api-access-v4pcs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:28.210334 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.210328 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62cq\" (UniqueName: \"kubernetes.io/projected/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5-kube-api-access-k62cq\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:28.233105 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.233024 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3198c99-d510-4e1e-bd02-db19e14a4884" path="/var/lib/kubelet/pods/e3198c99-d510-4e1e-bd02-db19e14a4884/volumes" Sep 10 15:03:28.279138 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.279106 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:28.989534 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.989508 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" Sep 10 15:03:28.989534 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.989518 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb" event={"ID":"4683ba2b-3433-475c-a17f-2a31930df65a","Type":"ContainerDied","Data":"00eeddc804a0e8d5fac0de0a4f50ccf0a0508d370447cbe4bb042ba1c1839006"} Sep 10 15:03:28.990052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.989594 2581 scope.go:117] "RemoveContainer" containerID="f6cba88a6ed6973a38b6ecf91e9e516e512e2e804c7a3fc918849848099c19af" Sep 10 15:03:28.991187 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.991167 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" Sep 10 15:03:28.991312 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:28.991211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr" event={"ID":"c30ffa16-cf70-49fe-8c3f-930e95a4f0c5","Type":"ContainerDied","Data":"f03c9bad890cbd8fd51500c5aa020317b375c8d23070f538b9f38d71f60de8dd"} Sep 10 15:03:29.001532 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:29.001509 2581 scope.go:117] "RemoveContainer" containerID="f8115fe6930d68ad2e2b2e8d4936b11b48625c6b3ed5c6befda179b43e442ca0" Sep 10 15:03:29.014130 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:29.014099 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:03:29.020722 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:29.020696 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c75e68-cb6c4c7c8-dbvkb"] Sep 10 15:03:29.034503 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:29.034476 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:03:29.039786 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:29.039733 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c75e68-6c57c4c7-hg6tr"] Sep 10 15:03:30.231240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:30.231203 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4683ba2b-3433-475c-a17f-2a31930df65a" path="/var/lib/kubelet/pods/4683ba2b-3433-475c-a17f-2a31930df65a/volumes" Sep 10 15:03:30.231612 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:30.231601 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" path="/var/lib/kubelet/pods/c30ffa16-cf70-49fe-8c3f-930e95a4f0c5/volumes" Sep 10 15:03:33.349004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:33.348963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:38.993553 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:38.993519 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:38.994354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:38.994334 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:46.547484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:46.547437 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:49.381165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381128 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt"] Sep 10 15:03:49.381611 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381595 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" containerName="payload-processing" Sep 10 15:03:49.381651 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381614 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" containerName="payload-processing" Sep 10 15:03:49.381651 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381635 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4683ba2b-3433-475c-a17f-2a31930df65a" containerName="payload-pre-processing" Sep 10 15:03:49.381651 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381641 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4683ba2b-3433-475c-a17f-2a31930df65a" containerName="payload-pre-processing" Sep 10 15:03:49.381778 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381729 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30ffa16-cf70-49fe-8c3f-930e95a4f0c5" containerName="payload-processing" Sep 10 15:03:49.381778 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.381748 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4683ba2b-3433-475c-a17f-2a31930df65a" containerName="payload-pre-processing" Sep 10 15:03:49.384973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.384956 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:49.387563 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.387545 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:03:49.398778 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.394434 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt"] Sep 10 15:03:49.401126 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.400287 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt\" (UID: \"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:49.501195 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.501150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt\" (UID: \"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:49.501963 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.501939 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt\" (UID: \"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:49.706240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.706148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:49.879972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.879936 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:49.987296 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:49.987269 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt"] Sep 10 15:03:49.990519 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:03:49.990479 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3ff103_0f31_42ba_9c6c_5368e2b8a4a7.slice/crio-2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4 WatchSource:0}: Error finding container 2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4: Status 404 returned error can't find the container with id 2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4 Sep 10 15:03:50.120282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:50.120248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" event={"ID":"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7","Type":"ContainerStarted","Data":"2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4"} Sep 10 15:03:51.127131 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:51.127094 2581 generic.go:356] "Generic (PLEG): container finished" podID="3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" containerID="3c9fed00efe83fc7aa92c1c7a546674ecd03c64b7477f378fb4f29acefd7902d" exitCode=0 Sep 10 15:03:51.127686 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:51.127166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" event={"ID":"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7","Type":"ContainerDied","Data":"3c9fed00efe83fc7aa92c1c7a546674ecd03c64b7477f378fb4f29acefd7902d"} Sep 10 15:03:52.285156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:52.285128 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:52.322466 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:52.322436 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca\") pod \"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7\" (UID: \"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7\") " Sep 10 15:03:52.322813 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:52.322789 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" (UID: "3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:52.423618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:52.423535 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:53.139806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:53.139743 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" Sep 10 15:03:53.139806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:53.139790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt" event={"ID":"3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7","Type":"ContainerDied","Data":"2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4"} Sep 10 15:03:53.140090 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:53.139834 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bfe9f566ea1c1e4ead9a81e78607930d688e8310c5ac5dce004785f18bd6fb4" Sep 10 15:03:56.723468 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.723421 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:56.771394 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.771321 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:56.771666 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.771636 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" podUID="f44545ed-914e-448f-bf9c-19c19209d9de" containerName="payload-processing" containerID="cri-o://2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d" gracePeriod=30 Sep 10 15:03:56.779410 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.779379 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:56.779644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.779612 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" podUID="1d984c75-181a-4816-8162-f1d2eb091a41" containerName="payload-pre-processing" containerID="cri-o://fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957" gracePeriod=30 Sep 10 15:03:56.827688 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.827653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:56.921738 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:56.921080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:57.002612 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.002568 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt"] Sep 10 15:03:57.006113 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.006046 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c3f76b1a-2vsvt"] Sep 10 15:03:57.010251 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.010219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:57.074827 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.074793 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:57.077599 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.077580 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:57.160338 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.160302 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8gcp\" (UniqueName: \"kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp\") pod \"f44545ed-914e-448f-bf9c-19c19209d9de\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " Sep 10 15:03:57.160549 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.160399 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume\") pod \"f44545ed-914e-448f-bf9c-19c19209d9de\" (UID: \"f44545ed-914e-448f-bf9c-19c19209d9de\") " Sep 10 15:03:57.160549 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.160457 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mj4v\" (UniqueName: \"kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v\") pod \"1d984c75-181a-4816-8162-f1d2eb091a41\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " Sep 10 15:03:57.160549 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.160484 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume\") pod \"1d984c75-181a-4816-8162-f1d2eb091a41\" (UID: \"1d984c75-181a-4816-8162-f1d2eb091a41\") " Sep 10 15:03:57.161300 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.161267 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f44545ed-914e-448f-bf9c-19c19209d9de" (UID: "f44545ed-914e-448f-bf9c-19c19209d9de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:57.161481 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.161443 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d984c75-181a-4816-8162-f1d2eb091a41" (UID: "1d984c75-181a-4816-8162-f1d2eb091a41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:57.164127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.164092 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp" (OuterVolumeSpecName: "kube-api-access-p8gcp") pod "f44545ed-914e-448f-bf9c-19c19209d9de" (UID: "f44545ed-914e-448f-bf9c-19c19209d9de"). InnerVolumeSpecName "kube-api-access-p8gcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:57.164805 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.164728 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v" (OuterVolumeSpecName: "kube-api-access-5mj4v") pod "1d984c75-181a-4816-8162-f1d2eb091a41" (UID: "1d984c75-181a-4816-8162-f1d2eb091a41"). InnerVolumeSpecName "kube-api-access-5mj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:57.165208 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.165184 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44545ed-914e-448f-bf9c-19c19209d9de" containerID="2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d" exitCode=0 Sep 10 15:03:57.165315 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.165293 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" Sep 10 15:03:57.165382 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.165328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" event={"ID":"f44545ed-914e-448f-bf9c-19c19209d9de","Type":"ContainerDied","Data":"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d"} Sep 10 15:03:57.165443 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.165396 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs" event={"ID":"f44545ed-914e-448f-bf9c-19c19209d9de","Type":"ContainerDied","Data":"4f049351c12314e396ac2767151efca787fa18e36ff11921809208b7f1f3fef9"} Sep 10 15:03:57.165443 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.165431 2581 scope.go:117] "RemoveContainer" containerID="2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d" Sep 10 15:03:57.168277 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.168184 2581 generic.go:356] "Generic (PLEG): container finished" podID="1d984c75-181a-4816-8162-f1d2eb091a41" containerID="fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957" exitCode=0 Sep 10 15:03:57.168277 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.168244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" event={"ID":"1d984c75-181a-4816-8162-f1d2eb091a41","Type":"ContainerDied","Data":"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957"} Sep 10 15:03:57.168458 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.168296 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" event={"ID":"1d984c75-181a-4816-8162-f1d2eb091a41","Type":"ContainerDied","Data":"a15a1b8b79916911edb43d7c072dd1c20bafc2f0b8a0f0c9583fdadee31ae9b5"} Sep 10 15:03:57.168609 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.168560 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw" Sep 10 15:03:57.183331 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.183308 2581 scope.go:117] "RemoveContainer" containerID="2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d" Sep 10 15:03:57.183707 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:03:57.183684 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d\": container with ID starting with 2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d not found: ID does not exist" containerID="2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d" Sep 10 15:03:57.183830 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.183721 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d"} err="failed to get container status \"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d\": rpc error: code = NotFound desc = could not find container \"2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d\": container with ID starting with 2f1dc03b6d669a3c1c14b362662f87c684426746d22059c3860ee98ca423f09d not found: ID does not exist" Sep 10 15:03:57.183830 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.183745 2581 scope.go:117] "RemoveContainer" containerID="fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957" Sep 10 15:03:57.197928 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.197870 2581 scope.go:117] "RemoveContainer" containerID="fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957" Sep 10 15:03:57.198332 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:03:57.198307 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957\": container with ID starting with fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957 not found: ID does not exist" containerID="fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957" Sep 10 15:03:57.198449 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.198343 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957"} err="failed to get container status \"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957\": rpc error: code = NotFound desc = could not find container \"fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957\": container with ID starting with fc11c3cc5b6afba6550ad9ab6d8dc2f5f04e3f2fc3cb57fdbd9221f6fc09a957 not found: ID does not exist" Sep 10 15:03:57.198449 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.198375 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:57.202058 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.201988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:57.202662 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.202633 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c3f76b1a-58b69bf577-zqgjs"] Sep 10 15:03:57.214915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.214884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:57.219912 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.219886 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c3f76b1a-67f86b97d6-gx4gw"] Sep 10 15:03:57.261513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.261432 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mj4v\" (UniqueName: \"kubernetes.io/projected/1d984c75-181a-4816-8162-f1d2eb091a41-kube-api-access-5mj4v\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.261513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.261460 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d984c75-181a-4816-8162-f1d2eb091a41-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.261513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.261471 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gcp\" (UniqueName: \"kubernetes.io/projected/f44545ed-914e-448f-bf9c-19c19209d9de-kube-api-access-p8gcp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.261513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.261481 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44545ed-914e-448f-bf9c-19c19209d9de-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.925801 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:57.925746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:58.054472 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:58.054436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:03:58.231833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:58.231715 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d984c75-181a-4816-8162-f1d2eb091a41" path="/var/lib/kubelet/pods/1d984c75-181a-4816-8162-f1d2eb091a41/volumes" Sep 10 15:03:58.232129 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:58.232113 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" path="/var/lib/kubelet/pods/3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7/volumes" Sep 10 15:03:58.232448 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:03:58.232436 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44545ed-914e-448f-bf9c-19c19209d9de" path="/var/lib/kubelet/pods/f44545ed-914e-448f-bf9c-19c19209d9de/volumes" Sep 10 15:04:09.684070 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:09.684032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:11.051947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:11.051908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:12.763073 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:12.763034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:16.547677 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:16.547638 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:16.722295 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:16.722243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:16.861332 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:16.861237 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:17.248423 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:17.248391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:17.948682 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:17.948646 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:18.953505 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:18.953470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:19.191810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:19.191776 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:19.646465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:19.646432 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:19.950015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:19.949913 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:20.391048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:20.391009 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:23.565885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:23.565836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:23.663033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:23.662992 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:24.757626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:24.757593 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:24.835425 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:24.835396 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:25.270567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.270518 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:25.575600 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.575516 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:25.672849 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.672815 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:25.697157 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697122 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:04:25.697596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697578 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44545ed-914e-448f-bf9c-19c19209d9de" containerName="payload-processing" Sep 10 15:04:25.697689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697600 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44545ed-914e-448f-bf9c-19c19209d9de" containerName="payload-processing" Sep 10 15:04:25.697689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697624 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d984c75-181a-4816-8162-f1d2eb091a41" containerName="payload-pre-processing" Sep 10 15:04:25.697689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697633 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d984c75-181a-4816-8162-f1d2eb091a41" containerName="payload-pre-processing" Sep 10 15:04:25.697689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697644 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" containerName="revoke-keys" Sep 10 15:04:25.697689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697653 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" containerName="revoke-keys" Sep 10 15:04:25.697972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697782 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d984c75-181a-4816-8162-f1d2eb091a41" containerName="payload-pre-processing" Sep 10 15:04:25.697972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697796 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3ff103-0f31-42ba-9c6c-5368e2b8a4a7" containerName="revoke-keys" Sep 10 15:04:25.697972 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.697808 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44545ed-914e-448f-bf9c-19c19209d9de" containerName="payload-processing" Sep 10 15:04:25.701427 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.701396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.704387 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.704364 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-78f25d-dockercfg-wcwdq\"" Sep 10 15:04:25.704519 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.704391 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-78f25d\"" Sep 10 15:04:25.715542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.715516 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:04:25.757652 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.757609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:04:25.761363 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.761337 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:25.776676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.776644 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:04:25.830059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.829973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.830059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.830048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqj6\" (UniqueName: \"kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.931072 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.931027 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8wq\" (UniqueName: \"kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:25.931274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.931090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqj6\" (UniqueName: \"kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.931274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.931233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:25.931398 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.931283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.932156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.932133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:25.939928 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:25.939896 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqj6\" (UniqueName: \"kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6\") pod \"payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:26.012733 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.012699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:26.032522 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.032492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:26.032685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.032547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8wq\" (UniqueName: \"kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:26.033497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.033471 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:26.044657 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.044613 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8wq\" (UniqueName: \"kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq\") pod \"payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:26.073863 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.073823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:26.301176 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.301150 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:04:26.303142 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:04:26.303104 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb651a0_c562_4997_8635_fd9a096bcaac.slice/crio-e4822abc31cce38284b50a441890375bbc52c17749d7e14897f5142483dd72a3 WatchSource:0}: Error finding container e4822abc31cce38284b50a441890375bbc52c17749d7e14897f5142483dd72a3: Status 404 returned error can't find the container with id e4822abc31cce38284b50a441890375bbc52c17749d7e14897f5142483dd72a3 Sep 10 15:04:26.335201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.335162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" event={"ID":"fdb651a0-c562-4997-8635-fd9a096bcaac","Type":"ContainerStarted","Data":"e4822abc31cce38284b50a441890375bbc52c17749d7e14897f5142483dd72a3"} Sep 10 15:04:26.368922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:26.368653 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:04:26.370791 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:04:26.370728 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2ec7a4_35ee_4952_8365_b6ae17e8d66f.slice/crio-96f9be2d170173e66feccab18c0f44012e75fbe48fcf893de93eb8d113fa53fe WatchSource:0}: Error finding container 96f9be2d170173e66feccab18c0f44012e75fbe48fcf893de93eb8d113fa53fe: Status 404 returned error can't find the container with id 96f9be2d170173e66feccab18c0f44012e75fbe48fcf893de93eb8d113fa53fe Sep 10 15:04:27.342294 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.342255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" event={"ID":"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f","Type":"ContainerStarted","Data":"6be12f01b84c47c97c0eb2a65f92297ded4858675883f2a0da57360b428b5670"} Sep 10 15:04:27.342827 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.342434 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:27.342827 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.342496 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" event={"ID":"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f","Type":"ContainerStarted","Data":"96f9be2d170173e66feccab18c0f44012e75fbe48fcf893de93eb8d113fa53fe"} Sep 10 15:04:27.343925 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.343902 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" event={"ID":"fdb651a0-c562-4997-8635-fd9a096bcaac","Type":"ContainerStarted","Data":"716806e0f3c5e6ad1b0ba5cb420e5d0823f64ba177584c3f6f1a1a6c83376615"} Sep 10 15:04:27.344042 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.343952 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:27.359240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.359182 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" podStartSLOduration=2.35916667 podStartE2EDuration="2.35916667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:27.358477611 +0000 UTC m=+1021.652779763" watchObservedRunningTime="2026-09-10 15:04:27.35916667 +0000 UTC m=+1021.653468795" Sep 10 15:04:27.376936 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:27.376882 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" podStartSLOduration=2.37686345 podStartE2EDuration="2.37686345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:27.37548515 +0000 UTC m=+1021.669787279" watchObservedRunningTime="2026-09-10 15:04:27.37686345 +0000 UTC m=+1021.671165579" Sep 10 15:04:38.351923 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:38.351892 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:04:38.352293 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:38.351962 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:04:43.118787 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:43.118734 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:48.378294 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:48.378260 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:49.950061 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:49.950018 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:50.082692 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:50.082649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:50.190635 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:50.190601 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:04:59.711190 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:04:59.711154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:05.979043 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:05.978999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:07.611797 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.611728 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:07.703822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.703786 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz"] Sep 10 15:05:07.707052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.707032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:07.709691 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.709669 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:05:07.717804 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.717778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz"] Sep 10 15:05:07.795855 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.795808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz\" (UID: \"f0836e4d-f906-4edc-9cf7-471a375bc633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:07.896841 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.896720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz\" (UID: \"f0836e4d-f906-4edc-9cf7-471a375bc633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:07.897536 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:07.897512 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz\" (UID: \"f0836e4d-f906-4edc-9cf7-471a375bc633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:08.021806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:08.021747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:08.299356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:08.299330 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz"] Sep 10 15:05:08.302257 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:05:08.302229 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0836e4d_f906_4edc_9cf7_471a375bc633.slice/crio-07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291 WatchSource:0}: Error finding container 07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291: Status 404 returned error can't find the container with id 07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291 Sep 10 15:05:08.586883 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:08.586791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" event={"ID":"f0836e4d-f906-4edc-9cf7-471a375bc633","Type":"ContainerStarted","Data":"07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291"} Sep 10 15:05:09.593506 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:09.593469 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0836e4d-f906-4edc-9cf7-471a375bc633" containerID="4a90846da7d23aed4d1c07a25b57b948d5a5ee9d580237010727dd633996cbdd" exitCode=0 Sep 10 15:05:09.593937 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:09.593560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" event={"ID":"f0836e4d-f906-4edc-9cf7-471a375bc633","Type":"ContainerDied","Data":"4a90846da7d23aed4d1c07a25b57b948d5a5ee9d580237010727dd633996cbdd"} Sep 10 15:05:10.745061 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:10.745034 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:10.825817 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:10.825751 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca\") pod \"f0836e4d-f906-4edc-9cf7-471a375bc633\" (UID: \"f0836e4d-f906-4edc-9cf7-471a375bc633\") " Sep 10 15:05:10.826142 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:10.826121 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0836e4d-f906-4edc-9cf7-471a375bc633" (UID: "f0836e4d-f906-4edc-9cf7-471a375bc633"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:10.927584 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:10.927491 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0836e4d-f906-4edc-9cf7-471a375bc633-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:11.605413 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:11.605385 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" Sep 10 15:05:11.605595 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:11.605418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz" event={"ID":"f0836e4d-f906-4edc-9cf7-471a375bc633","Type":"ContainerDied","Data":"07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291"} Sep 10 15:05:11.605595 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:11.605468 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b8884ef30ec8a980068f2a3f97003c9b90ab9c088a9dff5d08ee2427601291" Sep 10 15:05:12.037079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:12.037046 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:12.155560 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:12.155523 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:12.247372 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:12.247336 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.372619 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.372526 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.431689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.431659 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:05:18.431921 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.431897 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" podUID="fdb651a0-c562-4997-8635-fd9a096bcaac" containerName="payload-processing" containerID="cri-o://716806e0f3c5e6ad1b0ba5cb420e5d0823f64ba177584c3f6f1a1a6c83376615" gracePeriod=30 Sep 10 15:05:18.445059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.445021 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:05:18.445271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.445249 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" podUID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" containerName="payload-pre-processing" containerID="cri-o://6be12f01b84c47c97c0eb2a65f92297ded4858675883f2a0da57360b428b5670" gracePeriod=30 Sep 10 15:05:18.484251 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.484219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.580681 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.580645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.646370 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.646284 2581 generic.go:356] "Generic (PLEG): container finished" podID="fdb651a0-c562-4997-8635-fd9a096bcaac" containerID="716806e0f3c5e6ad1b0ba5cb420e5d0823f64ba177584c3f6f1a1a6c83376615" exitCode=0 Sep 10 15:05:18.646521 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.646359 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" event={"ID":"fdb651a0-c562-4997-8635-fd9a096bcaac","Type":"ContainerDied","Data":"716806e0f3c5e6ad1b0ba5cb420e5d0823f64ba177584c3f6f1a1a6c83376615"} Sep 10 15:05:18.648063 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.648035 2581 generic.go:356] "Generic (PLEG): container finished" podID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" containerID="6be12f01b84c47c97c0eb2a65f92297ded4858675883f2a0da57360b428b5670" exitCode=0 Sep 10 15:05:18.648198 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.648108 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" event={"ID":"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f","Type":"ContainerDied","Data":"6be12f01b84c47c97c0eb2a65f92297ded4858675883f2a0da57360b428b5670"} Sep 10 15:05:18.680384 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.680350 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz"] Sep 10 15:05:18.682731 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.682682 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-78f25d-gqbwz"] Sep 10 15:05:18.683735 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.683696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.733454 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.733430 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:05:18.736561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.736540 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:05:18.758549 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.758516 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:18.793545 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.793506 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hqj6\" (UniqueName: \"kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6\") pod \"fdb651a0-c562-4997-8635-fd9a096bcaac\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " Sep 10 15:05:18.793741 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.793600 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume\") pod \"fdb651a0-c562-4997-8635-fd9a096bcaac\" (UID: \"fdb651a0-c562-4997-8635-fd9a096bcaac\") " Sep 10 15:05:18.793741 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.793658 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt8wq\" (UniqueName: \"kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq\") pod \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " Sep 10 15:05:18.793741 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.793700 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume\") pod \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\" (UID: \"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f\") " Sep 10 15:05:18.794328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.794296 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" (UID: "7b2ec7a4-35ee-4952-8365-b6ae17e8d66f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:18.794589 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.794561 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdb651a0-c562-4997-8635-fd9a096bcaac" (UID: "fdb651a0-c562-4997-8635-fd9a096bcaac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:18.797005 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.796978 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6" (OuterVolumeSpecName: "kube-api-access-8hqj6") pod "fdb651a0-c562-4997-8635-fd9a096bcaac" (UID: "fdb651a0-c562-4997-8635-fd9a096bcaac"). InnerVolumeSpecName "kube-api-access-8hqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:18.797112 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.797021 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq" (OuterVolumeSpecName: "kube-api-access-kt8wq") pod "7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" (UID: "7b2ec7a4-35ee-4952-8365-b6ae17e8d66f"). InnerVolumeSpecName "kube-api-access-kt8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:18.894772 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.894711 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hqj6\" (UniqueName: \"kubernetes.io/projected/fdb651a0-c562-4997-8635-fd9a096bcaac-kube-api-access-8hqj6\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.894772 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.894772 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb651a0-c562-4997-8635-fd9a096bcaac-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.895048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.894789 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8wq\" (UniqueName: \"kubernetes.io/projected/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-kube-api-access-kt8wq\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.895048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:18.894803 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:19.050444 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.050410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:19.655968 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.655937 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" Sep 10 15:05:19.655968 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.655947 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh" event={"ID":"fdb651a0-c562-4997-8635-fd9a096bcaac","Type":"ContainerDied","Data":"e4822abc31cce38284b50a441890375bbc52c17749d7e14897f5142483dd72a3"} Sep 10 15:05:19.656559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.656026 2581 scope.go:117] "RemoveContainer" containerID="716806e0f3c5e6ad1b0ba5cb420e5d0823f64ba177584c3f6f1a1a6c83376615" Sep 10 15:05:19.657438 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.657333 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" Sep 10 15:05:19.657438 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.657362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd" event={"ID":"7b2ec7a4-35ee-4952-8365-b6ae17e8d66f","Type":"ContainerDied","Data":"96f9be2d170173e66feccab18c0f44012e75fbe48fcf893de93eb8d113fa53fe"} Sep 10 15:05:19.665838 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.665816 2581 scope.go:117] "RemoveContainer" containerID="6be12f01b84c47c97c0eb2a65f92297ded4858675883f2a0da57360b428b5670" Sep 10 15:05:19.680240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.680209 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:05:19.686279 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.686249 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78f25d-7c5dd6ff8f-mm9nh"] Sep 10 15:05:19.699057 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.699023 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:05:19.702485 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:19.702452 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78f25d-97969dc46-cn2xd"] Sep 10 15:05:20.231161 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:20.231122 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" path="/var/lib/kubelet/pods/7b2ec7a4-35ee-4952-8365-b6ae17e8d66f/volumes" Sep 10 15:05:20.231523 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:20.231506 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0836e4d-f906-4edc-9cf7-471a375bc633" path="/var/lib/kubelet/pods/f0836e4d-f906-4edc-9cf7-471a375bc633/volumes" Sep 10 15:05:20.231855 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:20.231841 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdb651a0-c562-4997-8635-fd9a096bcaac" path="/var/lib/kubelet/pods/fdb651a0-c562-4997-8635-fd9a096bcaac/volumes" Sep 10 15:05:22.660081 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:22.660051 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:23.395465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:23.395430 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:25.373070 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:25.373028 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:25.473134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:25.473096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:25.612967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:25.612926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:26.051656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:26.051614 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:26.388804 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:26.388746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:26.685565 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:26.685473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:28.208579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:28.208536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:28.301093 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:28.301054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:28.501927 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:28.501894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:29.822210 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:29.822168 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:29.934211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:29.934178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:30.024960 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:30.024920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:30.126916 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:30.126828 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:30.834523 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:30.834488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:30.948510 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:30.948460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:32.459402 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:32.459353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:38.183086 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:38.183042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:38.248971 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:38.248934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:38.843970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:38.843931 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:41.554815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:41.554786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:43.662464 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:43.662426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:44.977268 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:44.977227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:45.124711 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:45.124674 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:45.264008 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:45.263969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:45.572901 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:45.572786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:45.956986 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:45.956885 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:46.596793 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:46.596737 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:46.700630 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:46.700588 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:46.924639 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:46.924536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:47.034079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.034036 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:47.764591 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.764550 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:47.870659 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.870620 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc"] Sep 10 15:05:47.871123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871099 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0836e4d-f906-4edc-9cf7-471a375bc633" containerName="revoke-keys" Sep 10 15:05:47.871123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871124 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0836e4d-f906-4edc-9cf7-471a375bc633" containerName="revoke-keys" Sep 10 15:05:47.871282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871138 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" containerName="payload-pre-processing" Sep 10 15:05:47.871282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871147 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" containerName="payload-pre-processing" Sep 10 15:05:47.871282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871175 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb651a0-c562-4997-8635-fd9a096bcaac" containerName="payload-processing" Sep 10 15:05:47.871282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871183 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb651a0-c562-4997-8635-fd9a096bcaac" containerName="payload-processing" Sep 10 15:05:47.871282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871279 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0836e4d-f906-4edc-9cf7-471a375bc633" containerName="revoke-keys" Sep 10 15:05:47.871455 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871295 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2ec7a4-35ee-4952-8365-b6ae17e8d66f" containerName="payload-pre-processing" Sep 10 15:05:47.871455 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.871303 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb651a0-c562-4997-8635-fd9a096bcaac" containerName="payload-processing" Sep 10 15:05:47.875815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.875797 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:47.878260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.878240 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:05:47.886801 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.886710 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc"] Sep 10 15:05:47.950331 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.950291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc\" (UID: \"15df8999-31a1-41fe-9c32-cea8f0358592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:47.984978 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:47.984944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:48.051435 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.051337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc\" (UID: \"15df8999-31a1-41fe-9c32-cea8f0358592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:48.051965 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.051947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc\" (UID: \"15df8999-31a1-41fe-9c32-cea8f0358592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:48.189663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.189632 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:48.464253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.464229 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc"] Sep 10 15:05:48.466406 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:05:48.466381 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15df8999_31a1_41fe_9c32_cea8f0358592.slice/crio-bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad WatchSource:0}: Error finding container bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad: Status 404 returned error can't find the container with id bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad Sep 10 15:05:48.817248 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.817214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" event={"ID":"15df8999-31a1-41fe-9c32-cea8f0358592","Type":"ContainerStarted","Data":"e1fc3254b137a508d98282b1d8b92b12b90d12ec8c5a9acbd961f5d6965b80b4"} Sep 10 15:05:48.817610 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.817272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" event={"ID":"15df8999-31a1-41fe-9c32-cea8f0358592","Type":"ContainerStarted","Data":"bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad"} Sep 10 15:05:48.833380 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.833329 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" podStartSLOduration=1.833312514 podStartE2EDuration="1.833312514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:48.832070498 +0000 UTC m=+1103.126372627" watchObservedRunningTime="2026-09-10 15:05:48.833312514 +0000 UTC m=+1103.127614644" Sep 10 15:05:48.995187 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:48.995148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:49.087062 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:49.086952 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:49.824670 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:49.824638 2581 generic.go:356] "Generic (PLEG): container finished" podID="15df8999-31a1-41fe-9c32-cea8f0358592" containerID="e1fc3254b137a508d98282b1d8b92b12b90d12ec8c5a9acbd961f5d6965b80b4" exitCode=0 Sep 10 15:05:49.825141 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:49.824785 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" event={"ID":"15df8999-31a1-41fe-9c32-cea8f0358592","Type":"ContainerDied","Data":"e1fc3254b137a508d98282b1d8b92b12b90d12ec8c5a9acbd961f5d6965b80b4"} Sep 10 15:05:50.135733 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.135641 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:50.238873 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.238840 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:50.338019 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.337979 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:50.613706 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.613671 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:50.691074 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.691020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:50.744736 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.744698 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:05:50.748354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.748337 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.754380 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.754357 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-411a8086-dockercfg-55z74\"" Sep 10 15:05:50.754798 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.754777 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-411a8086\"" Sep 10 15:05:50.771028 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.770994 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:05:50.774843 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.774807 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:05:50.774967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.774838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.774967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.774940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87whm\" (UniqueName: \"kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.778874 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.778849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.790271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.790240 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:05:50.876124 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.876036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.876124 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.876091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.876525 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.876147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5ng\" (UniqueName: \"kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.876525 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.876167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87whm\" (UniqueName: \"kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.876866 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.876841 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.886886 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.886682 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87whm\" (UniqueName: \"kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm\") pod \"payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:50.968611 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.968582 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:50.977403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.977371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5ng\" (UniqueName: \"kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.977559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.977500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.978377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.978350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:50.989059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:50.989029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5ng\" (UniqueName: \"kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng\") pod \"payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:51.059040 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.059005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:51.078460 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.078434 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca\") pod \"15df8999-31a1-41fe-9c32-cea8f0358592\" (UID: \"15df8999-31a1-41fe-9c32-cea8f0358592\") " Sep 10 15:05:51.078784 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.078746 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15df8999-31a1-41fe-9c32-cea8f0358592" (UID: "15df8999-31a1-41fe-9c32-cea8f0358592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:51.089409 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.089383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:51.179662 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.179622 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8999-31a1-41fe-9c32-cea8f0358592-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:51.340263 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.340237 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:05:51.342727 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:05:51.342691 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d396fa_2f66_4e65_a81d_77a0818880a8.slice/crio-9904416a52c8ad5ae1ae8b08eca16099f83597eb70bbb0c9f525b0a3bdf01dd9 WatchSource:0}: Error finding container 9904416a52c8ad5ae1ae8b08eca16099f83597eb70bbb0c9f525b0a3bdf01dd9: Status 404 returned error can't find the container with id 9904416a52c8ad5ae1ae8b08eca16099f83597eb70bbb0c9f525b0a3bdf01dd9 Sep 10 15:05:51.374998 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.374870 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:05:51.840257 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.840227 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" Sep 10 15:05:51.840441 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.840257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc" event={"ID":"15df8999-31a1-41fe-9c32-cea8f0358592","Type":"ContainerDied","Data":"bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad"} Sep 10 15:05:51.840441 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.840313 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd18f5f412ba499112a77d603c30804f1eb30ef88f4b4b46dcc6a4bb0d8023ad" Sep 10 15:05:51.842243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.842105 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" event={"ID":"754c410c-c882-490a-ba00-72ab36a5af9f","Type":"ContainerStarted","Data":"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2"} Sep 10 15:05:51.842243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.842159 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" event={"ID":"754c410c-c882-490a-ba00-72ab36a5af9f","Type":"ContainerStarted","Data":"a0a9e884e1aad99c6fca3ce6fd4cab6848163065cdb36d3471dbf586529c41eb"} Sep 10 15:05:51.842243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.842202 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:05:51.843936 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.843898 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" event={"ID":"22d396fa-2f66-4e65-a81d-77a0818880a8","Type":"ContainerStarted","Data":"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70"} Sep 10 15:05:51.844079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.843941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" event={"ID":"22d396fa-2f66-4e65-a81d-77a0818880a8","Type":"ContainerStarted","Data":"9904416a52c8ad5ae1ae8b08eca16099f83597eb70bbb0c9f525b0a3bdf01dd9"} Sep 10 15:05:51.844079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.843961 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:05:51.860117 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.860057 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" podStartSLOduration=1.860033743 podStartE2EDuration="1.860033743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:51.858422246 +0000 UTC m=+1106.152724376" watchObservedRunningTime="2026-09-10 15:05:51.860033743 +0000 UTC m=+1106.154335875" Sep 10 15:05:51.877047 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:51.876992 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" podStartSLOduration=1.87697464 podStartE2EDuration="1.87697464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:51.876134614 +0000 UTC m=+1106.170436768" watchObservedRunningTime="2026-09-10 15:05:51.87697464 +0000 UTC m=+1106.171276769" Sep 10 15:05:52.045791 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:52.045721 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:57.859238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:57.859198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.016984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.016942 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.628977 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.628930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.675738 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.675706 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:05:58.676049 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.675993 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" podUID="deb8fbf8-4844-4865-867a-d9441134689e" containerName="payload-processing" containerID="cri-o://3b0e55ce5bff608aa63f5ab0508f7197bb96285877054b2efb465477c8d6f512" gracePeriod=30 Sep 10 15:05:58.679055 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.679034 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:05:58.679292 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.679246 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" podUID="6759748b-e2af-4e03-b648-809fdabc2e84" containerName="payload-pre-processing" containerID="cri-o://345e515d9791754d162188cbb0c5c7f33f56f01476304955c8018dbf0a53f1eb" gracePeriod=30 Sep 10 15:05:58.729087 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.729048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.802192 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.802160 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.860260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.860233 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc"] Sep 10 15:05:58.863813 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.863785 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e577f0-r2jvc"] Sep 10 15:05:58.871954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.871926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:58.891322 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.891289 2581 generic.go:356] "Generic (PLEG): container finished" podID="deb8fbf8-4844-4865-867a-d9441134689e" containerID="3b0e55ce5bff608aa63f5ab0508f7197bb96285877054b2efb465477c8d6f512" exitCode=0 Sep 10 15:05:58.891459 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.891324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" event={"ID":"deb8fbf8-4844-4865-867a-d9441134689e","Type":"ContainerDied","Data":"3b0e55ce5bff608aa63f5ab0508f7197bb96285877054b2efb465477c8d6f512"} Sep 10 15:05:58.894419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.894219 2581 generic.go:356] "Generic (PLEG): container finished" podID="6759748b-e2af-4e03-b648-809fdabc2e84" containerID="345e515d9791754d162188cbb0c5c7f33f56f01476304955c8018dbf0a53f1eb" exitCode=0 Sep 10 15:05:58.894419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.894360 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" event={"ID":"6759748b-e2af-4e03-b648-809fdabc2e84","Type":"ContainerDied","Data":"345e515d9791754d162188cbb0c5c7f33f56f01476304955c8018dbf0a53f1eb"} Sep 10 15:05:58.973535 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.973509 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:05:58.975596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:58.975578 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:05:59.052973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.052940 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnqw7\" (UniqueName: \"kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7\") pod \"6759748b-e2af-4e03-b648-809fdabc2e84\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " Sep 10 15:05:59.053152 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.052984 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmhcp\" (UniqueName: \"kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp\") pod \"deb8fbf8-4844-4865-867a-d9441134689e\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " Sep 10 15:05:59.053152 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.053006 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume\") pod \"6759748b-e2af-4e03-b648-809fdabc2e84\" (UID: \"6759748b-e2af-4e03-b648-809fdabc2e84\") " Sep 10 15:05:59.053152 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.053042 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume\") pod \"deb8fbf8-4844-4865-867a-d9441134689e\" (UID: \"deb8fbf8-4844-4865-867a-d9441134689e\") " Sep 10 15:05:59.053405 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.053380 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6759748b-e2af-4e03-b648-809fdabc2e84" (UID: "6759748b-e2af-4e03-b648-809fdabc2e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:59.053526 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.053410 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "deb8fbf8-4844-4865-867a-d9441134689e" (UID: "deb8fbf8-4844-4865-867a-d9441134689e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:59.055225 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.055201 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7" (OuterVolumeSpecName: "kube-api-access-fnqw7") pod "6759748b-e2af-4e03-b648-809fdabc2e84" (UID: "6759748b-e2af-4e03-b648-809fdabc2e84"). InnerVolumeSpecName "kube-api-access-fnqw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:59.055299 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.055253 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp" (OuterVolumeSpecName: "kube-api-access-rmhcp") pod "deb8fbf8-4844-4865-867a-d9441134689e" (UID: "deb8fbf8-4844-4865-867a-d9441134689e"). InnerVolumeSpecName "kube-api-access-rmhcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:59.154820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.154691 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqw7\" (UniqueName: \"kubernetes.io/projected/6759748b-e2af-4e03-b648-809fdabc2e84-kube-api-access-fnqw7\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.154820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.154730 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhcp\" (UniqueName: \"kubernetes.io/projected/deb8fbf8-4844-4865-867a-d9441134689e-kube-api-access-rmhcp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.154820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.154739 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6759748b-e2af-4e03-b648-809fdabc2e84-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.154820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.154750 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb8fbf8-4844-4865-867a-d9441134689e-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.210284 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.210248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:05:59.901033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.901002 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" Sep 10 15:05:59.901033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.901030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7" event={"ID":"6759748b-e2af-4e03-b648-809fdabc2e84","Type":"ContainerDied","Data":"55ea4afa89bbeb8fd2a37baa505f636e675531d83efdb5049db1a24dcd33a05a"} Sep 10 15:05:59.901545 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.901099 2581 scope.go:117] "RemoveContainer" containerID="345e515d9791754d162188cbb0c5c7f33f56f01476304955c8018dbf0a53f1eb" Sep 10 15:05:59.902685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.902660 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" Sep 10 15:05:59.902807 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.902655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr" event={"ID":"deb8fbf8-4844-4865-867a-d9441134689e","Type":"ContainerDied","Data":"5872b120a209669ecab338119f3e81c4bf98839b8b3d5acbd2cd57b6e6e1234a"} Sep 10 15:05:59.913517 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.913411 2581 scope.go:117] "RemoveContainer" containerID="3b0e55ce5bff608aa63f5ab0508f7197bb96285877054b2efb465477c8d6f512" Sep 10 15:05:59.932377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.932346 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:05:59.940884 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.940855 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e577f0-b766b4566-6qdp7"] Sep 10 15:05:59.959150 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.959120 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:05:59.964031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:05:59.964010 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e577f0-74d668ccf7-j8qdr"] Sep 10 15:06:00.230073 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:00.229993 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15df8999-31a1-41fe-9c32-cea8f0358592" path="/var/lib/kubelet/pods/15df8999-31a1-41fe-9c32-cea8f0358592/volumes" Sep 10 15:06:00.230342 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:00.230330 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6759748b-e2af-4e03-b648-809fdabc2e84" path="/var/lib/kubelet/pods/6759748b-e2af-4e03-b648-809fdabc2e84/volumes" Sep 10 15:06:00.230644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:00.230633 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb8fbf8-4844-4865-867a-d9441134689e" path="/var/lib/kubelet/pods/deb8fbf8-4844-4865-867a-d9441134689e/volumes" Sep 10 15:06:02.854874 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:02.854845 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:06:02.855441 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:02.854924 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:06:05.120205 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.120169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:05.204436 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.204404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:05.383550 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.383462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:05.696708 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.696622 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh"] Sep 10 15:06:05.697162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697143 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb8fbf8-4844-4865-867a-d9441134689e" containerName="payload-processing" Sep 10 15:06:05.697162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697165 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb8fbf8-4844-4865-867a-d9441134689e" containerName="payload-processing" Sep 10 15:06:05.697329 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697179 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6759748b-e2af-4e03-b648-809fdabc2e84" containerName="payload-pre-processing" Sep 10 15:06:05.697329 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697187 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6759748b-e2af-4e03-b648-809fdabc2e84" containerName="payload-pre-processing" Sep 10 15:06:05.697329 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697202 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15df8999-31a1-41fe-9c32-cea8f0358592" containerName="revoke-keys" Sep 10 15:06:05.697329 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697212 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15df8999-31a1-41fe-9c32-cea8f0358592" containerName="revoke-keys" Sep 10 15:06:05.697577 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697337 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15df8999-31a1-41fe-9c32-cea8f0358592" containerName="revoke-keys" Sep 10 15:06:05.697577 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697354 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6759748b-e2af-4e03-b648-809fdabc2e84" containerName="payload-pre-processing" Sep 10 15:06:05.697577 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.697366 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb8fbf8-4844-4865-867a-d9441134689e" containerName="payload-processing" Sep 10 15:06:05.702043 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.702020 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:05.704555 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.704536 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:06:05.715965 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.715941 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh"] Sep 10 15:06:05.812095 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.812051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh\" (UID: \"d44e0bc4-9837-492f-996c-0ac946302976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:05.909628 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.909587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:05.913586 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.913553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh\" (UID: \"d44e0bc4-9837-492f-996c-0ac946302976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:05.914187 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:05.914166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh\" (UID: \"d44e0bc4-9837-492f-996c-0ac946302976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:06.012890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:06.012851 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:06.284784 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:06.284742 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh"] Sep 10 15:06:06.286915 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:06.286884 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44e0bc4_9837_492f_996c_0ac946302976.slice/crio-70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc WatchSource:0}: Error finding container 70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc: Status 404 returned error can't find the container with id 70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc Sep 10 15:06:06.945006 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:06.944913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" event={"ID":"d44e0bc4-9837-492f-996c-0ac946302976","Type":"ContainerStarted","Data":"b8cdaf75bc99b9ae91acd1dc64c5929c7a1b7d8106a0e8d55a78efa4a9d67968"} Sep 10 15:06:06.945006 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:06.944963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" event={"ID":"d44e0bc4-9837-492f-996c-0ac946302976","Type":"ContainerStarted","Data":"70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc"} Sep 10 15:06:06.962161 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:06.962107 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" podStartSLOduration=1.962093273 podStartE2EDuration="1.962093273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:06.959707415 +0000 UTC m=+1121.254009543" watchObservedRunningTime="2026-09-10 15:06:06.962093273 +0000 UTC m=+1121.256395449" Sep 10 15:06:07.952855 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:07.952823 2581 generic.go:356] "Generic (PLEG): container finished" podID="d44e0bc4-9837-492f-996c-0ac946302976" containerID="b8cdaf75bc99b9ae91acd1dc64c5929c7a1b7d8106a0e8d55a78efa4a9d67968" exitCode=0 Sep 10 15:06:07.953246 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:07.952924 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" event={"ID":"d44e0bc4-9837-492f-996c-0ac946302976","Type":"ContainerDied","Data":"b8cdaf75bc99b9ae91acd1dc64c5929c7a1b7d8106a0e8d55a78efa4a9d67968"} Sep 10 15:06:09.100418 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.100386 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:09.248924 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.248892 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca\") pod \"d44e0bc4-9837-492f-996c-0ac946302976\" (UID: \"d44e0bc4-9837-492f-996c-0ac946302976\") " Sep 10 15:06:09.249242 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.249219 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d44e0bc4-9837-492f-996c-0ac946302976" (UID: "d44e0bc4-9837-492f-996c-0ac946302976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:09.350210 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.350174 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44e0bc4-9837-492f-996c-0ac946302976-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:09.965465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.965430 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" event={"ID":"d44e0bc4-9837-492f-996c-0ac946302976","Type":"ContainerDied","Data":"70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc"} Sep 10 15:06:09.965465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.965480 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70bcddc1dccbcee3f2bbd8726e24bf48190b819d26941edd88219b56afcce8fc" Sep 10 15:06:09.965692 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:09.965448 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh" Sep 10 15:06:11.054196 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:11.054163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:13.835038 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:13.835005 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:14.494690 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:14.494653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:14.590983 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:14.590949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:14.928053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:14.927952 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:15.818676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:15.818633 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:16.449131 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.449094 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:16.470406 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.470374 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:06:16.470661 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.470635 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" podUID="1af0ecce-73dc-4ca6-a272-67729798eb1b" containerName="payload-processing" containerID="cri-o://3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f" gracePeriod=30 Sep 10 15:06:16.482887 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.480768 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:06:16.482887 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.481142 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" podUID="66af9654-f0d0-4200-a47c-82594e91c352" containerName="payload-pre-processing" containerID="cri-o://481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954" gracePeriod=30 Sep 10 15:06:16.548865 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.548727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:16.635503 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.635471 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh"] Sep 10 15:06:16.638360 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.638297 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6356e4-bp6jh"] Sep 10 15:06:16.639155 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.639131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:16.703292 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.703216 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:16.751554 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.751527 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:06:16.777165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.777133 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:06:16.815849 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.815822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume\") pod \"1af0ecce-73dc-4ca6-a272-67729798eb1b\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " Sep 10 15:06:16.816003 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.815876 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2stvp\" (UniqueName: \"kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp\") pod \"1af0ecce-73dc-4ca6-a272-67729798eb1b\" (UID: \"1af0ecce-73dc-4ca6-a272-67729798eb1b\") " Sep 10 15:06:16.816272 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.816246 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1af0ecce-73dc-4ca6-a272-67729798eb1b" (UID: "1af0ecce-73dc-4ca6-a272-67729798eb1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:16.818079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.818056 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp" (OuterVolumeSpecName: "kube-api-access-2stvp") pod "1af0ecce-73dc-4ca6-a272-67729798eb1b" (UID: "1af0ecce-73dc-4ca6-a272-67729798eb1b"). InnerVolumeSpecName "kube-api-access-2stvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:16.917310 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.917268 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n72t7\" (UniqueName: \"kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7\") pod \"66af9654-f0d0-4200-a47c-82594e91c352\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " Sep 10 15:06:16.917520 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.917342 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume\") pod \"66af9654-f0d0-4200-a47c-82594e91c352\" (UID: \"66af9654-f0d0-4200-a47c-82594e91c352\") " Sep 10 15:06:16.917620 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.917600 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stvp\" (UniqueName: \"kubernetes.io/projected/1af0ecce-73dc-4ca6-a272-67729798eb1b-kube-api-access-2stvp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:16.917684 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.917627 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af0ecce-73dc-4ca6-a272-67729798eb1b-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:16.917684 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.917658 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66af9654-f0d0-4200-a47c-82594e91c352" (UID: "66af9654-f0d0-4200-a47c-82594e91c352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:16.919452 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:16.919431 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7" (OuterVolumeSpecName: "kube-api-access-n72t7") pod "66af9654-f0d0-4200-a47c-82594e91c352" (UID: "66af9654-f0d0-4200-a47c-82594e91c352"). InnerVolumeSpecName "kube-api-access-n72t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:17.003553 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.003521 2581 generic.go:356] "Generic (PLEG): container finished" podID="66af9654-f0d0-4200-a47c-82594e91c352" containerID="481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954" exitCode=0 Sep 10 15:06:17.003736 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.003569 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" Sep 10 15:06:17.003736 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.003609 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" event={"ID":"66af9654-f0d0-4200-a47c-82594e91c352","Type":"ContainerDied","Data":"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954"} Sep 10 15:06:17.003736 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.003668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j" event={"ID":"66af9654-f0d0-4200-a47c-82594e91c352","Type":"ContainerDied","Data":"097efd98cc7799313891d6f612e4b359e79b879c022ad805e9a8b6aafc6f3cb9"} Sep 10 15:06:17.003736 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.003699 2581 scope.go:117] "RemoveContainer" containerID="481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954" Sep 10 15:06:17.005278 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.005255 2581 generic.go:356] "Generic (PLEG): container finished" podID="1af0ecce-73dc-4ca6-a272-67729798eb1b" containerID="3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f" exitCode=0 Sep 10 15:06:17.005403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.005293 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" Sep 10 15:06:17.005403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.005350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" event={"ID":"1af0ecce-73dc-4ca6-a272-67729798eb1b","Type":"ContainerDied","Data":"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f"} Sep 10 15:06:17.005516 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.005403 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf" event={"ID":"1af0ecce-73dc-4ca6-a272-67729798eb1b","Type":"ContainerDied","Data":"12d410febb15f09a5970cae1adc5f309e5dc28acbb49533578bc0c5e3720a0cd"} Sep 10 15:06:17.014407 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.014389 2581 scope.go:117] "RemoveContainer" containerID="481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954" Sep 10 15:06:17.014701 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:06:17.014684 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954\": container with ID starting with 481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954 not found: ID does not exist" containerID="481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954" Sep 10 15:06:17.014816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.014708 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954"} err="failed to get container status \"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954\": rpc error: code = NotFound desc = could not find container \"481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954\": container with ID starting with 481e31f3b04b47fe61068c0e53953431f2398f31a45cc3ce95319bedecae9954 not found: ID does not exist" Sep 10 15:06:17.014816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.014722 2581 scope.go:117] "RemoveContainer" containerID="3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f" Sep 10 15:06:17.016907 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.016841 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:17.018219 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.018197 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72t7\" (UniqueName: \"kubernetes.io/projected/66af9654-f0d0-4200-a47c-82594e91c352-kube-api-access-n72t7\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:17.018290 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.018226 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af9654-f0d0-4200-a47c-82594e91c352-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:17.023937 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.023922 2581 scope.go:117] "RemoveContainer" containerID="3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f" Sep 10 15:06:17.024212 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:06:17.024193 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f\": container with ID starting with 3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f not found: ID does not exist" containerID="3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f" Sep 10 15:06:17.024253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.024218 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f"} err="failed to get container status \"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f\": rpc error: code = NotFound desc = could not find container \"3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f\": container with ID starting with 3e0de7a914f8df0aa1c522f4aeaacea4d758388ce4778d2b57d176ad73cf6e0f not found: ID does not exist" Sep 10 15:06:17.031242 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.031221 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:06:17.032898 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.032881 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6356e4-6bcb6486d9-2qmmf"] Sep 10 15:06:17.044809 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.044783 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:06:17.055240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:17.055219 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6356e4-55cf74bf74-6qw7j"] Sep 10 15:06:18.229969 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:18.229937 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af0ecce-73dc-4ca6-a272-67729798eb1b" path="/var/lib/kubelet/pods/1af0ecce-73dc-4ca6-a272-67729798eb1b/volumes" Sep 10 15:06:18.230343 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:18.230283 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66af9654-f0d0-4200-a47c-82594e91c352" path="/var/lib/kubelet/pods/66af9654-f0d0-4200-a47c-82594e91c352/volumes" Sep 10 15:06:18.230572 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:18.230559 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44e0bc4-9837-492f-996c-0ac946302976" path="/var/lib/kubelet/pods/d44e0bc4-9837-492f-996c-0ac946302976/volumes" Sep 10 15:06:20.752473 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:20.752433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:20.814488 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:20.814451 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:21.099086 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.098985 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x"] Sep 10 15:06:21.099566 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099546 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af0ecce-73dc-4ca6-a272-67729798eb1b" containerName="payload-processing" Sep 10 15:06:21.099656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099568 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af0ecce-73dc-4ca6-a272-67729798eb1b" containerName="payload-processing" Sep 10 15:06:21.099656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099598 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66af9654-f0d0-4200-a47c-82594e91c352" containerName="payload-pre-processing" Sep 10 15:06:21.099656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099607 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66af9654-f0d0-4200-a47c-82594e91c352" containerName="payload-pre-processing" Sep 10 15:06:21.099656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099621 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44e0bc4-9837-492f-996c-0ac946302976" containerName="revoke-keys" Sep 10 15:06:21.099656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099630 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44e0bc4-9837-492f-996c-0ac946302976" containerName="revoke-keys" Sep 10 15:06:21.099848 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099724 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af0ecce-73dc-4ca6-a272-67729798eb1b" containerName="payload-processing" Sep 10 15:06:21.099848 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099749 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44e0bc4-9837-492f-996c-0ac946302976" containerName="revoke-keys" Sep 10 15:06:21.099848 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.099780 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66af9654-f0d0-4200-a47c-82594e91c352" containerName="payload-pre-processing" Sep 10 15:06:21.104646 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.104627 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:21.107459 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.107438 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:06:21.111998 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.111968 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x"] Sep 10 15:06:21.252729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.252687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-411a8086-r4f4x\" (UID: \"6d43821a-9ec0-4bde-9d50-e69f12cceabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:21.292568 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.292535 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:21.353480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.353375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-411a8086-r4f4x\" (UID: \"6d43821a-9ec0-4bde-9d50-e69f12cceabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:21.354108 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.354083 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-411a8086-r4f4x\" (UID: \"6d43821a-9ec0-4bde-9d50-e69f12cceabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:21.417490 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.417457 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:21.706587 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:21.706559 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x"] Sep 10 15:06:21.708964 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:21.708938 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d43821a_9ec0_4bde_9d50_e69f12cceabd.slice/crio-07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f WatchSource:0}: Error finding container 07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f: Status 404 returned error can't find the container with id 07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f Sep 10 15:06:22.036199 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:22.036167 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" event={"ID":"6d43821a-9ec0-4bde-9d50-e69f12cceabd","Type":"ContainerStarted","Data":"07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f"} Sep 10 15:06:23.043090 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:23.043055 2581 generic.go:356] "Generic (PLEG): container finished" podID="6d43821a-9ec0-4bde-9d50-e69f12cceabd" containerID="67009177e524ddb00320c53bdaf13dd2c8abe0c614ee2d3fea670c0d2e5ebc9d" exitCode=0 Sep 10 15:06:23.043480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:23.043138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" event={"ID":"6d43821a-9ec0-4bde-9d50-e69f12cceabd","Type":"ContainerDied","Data":"67009177e524ddb00320c53bdaf13dd2c8abe0c614ee2d3fea670c0d2e5ebc9d"} Sep 10 15:06:23.260396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:23.260363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:23.412727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:23.412640 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:23.629621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:23.629589 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:24.054206 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.054176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:24.187031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.187009 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:24.280088 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.280054 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca\") pod \"6d43821a-9ec0-4bde-9d50-e69f12cceabd\" (UID: \"6d43821a-9ec0-4bde-9d50-e69f12cceabd\") " Sep 10 15:06:24.280467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.280440 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d43821a-9ec0-4bde-9d50-e69f12cceabd" (UID: "6d43821a-9ec0-4bde-9d50-e69f12cceabd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:24.381458 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.381356 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d43821a-9ec0-4bde-9d50-e69f12cceabd-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:24.397790 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.397626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:24.486602 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:24.486572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:25.055615 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:25.055583 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" Sep 10 15:06:25.056047 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:25.055616 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x" event={"ID":"6d43821a-9ec0-4bde-9d50-e69f12cceabd","Type":"ContainerDied","Data":"07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f"} Sep 10 15:06:25.056047 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:25.055691 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07cbc74ac30cee23b427d33687d002c39c690d326337fe3de7581def9b84574f" Sep 10 15:06:27.128062 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:27.128020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:27.226768 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:27.226720 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:28.383507 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:28.383472 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:28.516810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:28.516732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:28.974215 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:28.974181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:29.069578 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.069542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:29.161933 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.161890 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:29.162360 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.162340 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:06:29.162732 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.162718 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d43821a-9ec0-4bde-9d50-e69f12cceabd" containerName="revoke-keys" Sep 10 15:06:29.162732 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.162732 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d43821a-9ec0-4bde-9d50-e69f12cceabd" containerName="revoke-keys" Sep 10 15:06:29.162885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.162821 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d43821a-9ec0-4bde-9d50-e69f12cceabd" containerName="revoke-keys" Sep 10 15:06:29.166245 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.166223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.171710 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.171686 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-decc52\"" Sep 10 15:06:29.171906 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.171864 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-decc52-dockercfg-cg5bx\"" Sep 10 15:06:29.197154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.197117 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:06:29.205890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.205857 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:06:29.208488 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.208464 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.227707 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.227632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:06:29.321201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.321165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbv5\" (UniqueName: \"kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.321370 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.321220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhhr\" (UniqueName: \"kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.321370 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.321273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.321370 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.321296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.422093 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbv5\" (UniqueName: \"kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.422484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhhr\" (UniqueName: \"kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.422484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.422484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.422908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.422954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.422907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.430991 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.430961 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhhr\" (UniqueName: \"kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr\") pod \"payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.431115 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.430966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbv5\" (UniqueName: \"kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5\") pod \"payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.476979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.476943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:29.519969 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.519931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:29.754361 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.754336 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:06:29.755508 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:29.755483 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8953ab1_88fc_4a78_8d63_5d4ef64abc87.slice/crio-2b1f69fd197a7482bfa0db3caa2e47fc21be88aacd8b47de053908dd644b2625 WatchSource:0}: Error finding container 2b1f69fd197a7482bfa0db3caa2e47fc21be88aacd8b47de053908dd644b2625: Status 404 returned error can't find the container with id 2b1f69fd197a7482bfa0db3caa2e47fc21be88aacd8b47de053908dd644b2625 Sep 10 15:06:29.810106 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:29.810080 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:06:29.811653 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:29.811622 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482ff74_63e4_401d_80c5_4b4c1ea224a1.slice/crio-475cdafb813ed83935e66531d1a6e195b8201193b03164ec6a7f3483fdc32b81 WatchSource:0}: Error finding container 475cdafb813ed83935e66531d1a6e195b8201193b03164ec6a7f3483fdc32b81: Status 404 returned error can't find the container with id 475cdafb813ed83935e66531d1a6e195b8201193b03164ec6a7f3483fdc32b81 Sep 10 15:06:30.085479 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.085396 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" event={"ID":"d8953ab1-88fc-4a78-8d63-5d4ef64abc87","Type":"ContainerStarted","Data":"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1"} Sep 10 15:06:30.085479 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.085445 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" event={"ID":"d8953ab1-88fc-4a78-8d63-5d4ef64abc87","Type":"ContainerStarted","Data":"2b1f69fd197a7482bfa0db3caa2e47fc21be88aacd8b47de053908dd644b2625"} Sep 10 15:06:30.085703 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.085484 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:30.087134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.087106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" event={"ID":"3482ff74-63e4-401d-80c5-4b4c1ea224a1","Type":"ContainerStarted","Data":"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4"} Sep 10 15:06:30.087286 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.087154 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" event={"ID":"3482ff74-63e4-401d-80c5-4b4c1ea224a1","Type":"ContainerStarted","Data":"475cdafb813ed83935e66531d1a6e195b8201193b03164ec6a7f3483fdc32b81"} Sep 10 15:06:30.087286 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.087260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:30.102425 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.102374 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" podStartSLOduration=1.102352571 podStartE2EDuration="1.102352571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:30.101067876 +0000 UTC m=+1144.395370030" watchObservedRunningTime="2026-09-10 15:06:30.102352571 +0000 UTC m=+1144.396654700" Sep 10 15:06:30.119492 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.119424 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" podStartSLOduration=1.119408793 podStartE2EDuration="1.119408793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:30.117564632 +0000 UTC m=+1144.411866775" watchObservedRunningTime="2026-09-10 15:06:30.119408793 +0000 UTC m=+1144.413710921" Sep 10 15:06:30.477363 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:30.477273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.194128 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.194078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.222404 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.222368 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:06:31.222880 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.222629 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" podUID="754c410c-c882-490a-ba00-72ab36a5af9f" containerName="payload-processing" containerID="cri-o://85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2" gracePeriod=30 Sep 10 15:06:31.230947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.230912 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:06:31.231175 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.231148 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" podUID="22d396fa-2f66-4e65-a81d-77a0818880a8" containerName="payload-pre-processing" containerID="cri-o://bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70" gracePeriod=30 Sep 10 15:06:31.266919 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.266886 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.366714 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.366681 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.414813 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.414784 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x"] Sep 10 15:06:31.417861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.417832 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-411a8086-r4f4x"] Sep 10 15:06:31.435202 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.435121 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.516836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.516516 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:06:31.571265 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.571238 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:06:31.643436 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.643398 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume\") pod \"754c410c-c882-490a-ba00-72ab36a5af9f\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " Sep 10 15:06:31.643436 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.643436 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh5ng\" (UniqueName: \"kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng\") pod \"754c410c-c882-490a-ba00-72ab36a5af9f\" (UID: \"754c410c-c882-490a-ba00-72ab36a5af9f\") " Sep 10 15:06:31.643860 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.643837 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "754c410c-c882-490a-ba00-72ab36a5af9f" (UID: "754c410c-c882-490a-ba00-72ab36a5af9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:31.645797 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.645772 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng" (OuterVolumeSpecName: "kube-api-access-hh5ng") pod "754c410c-c882-490a-ba00-72ab36a5af9f" (UID: "754c410c-c882-490a-ba00-72ab36a5af9f"). InnerVolumeSpecName "kube-api-access-hh5ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:31.744771 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.744720 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume\") pod \"22d396fa-2f66-4e65-a81d-77a0818880a8\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " Sep 10 15:06:31.744946 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.744778 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87whm\" (UniqueName: \"kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm\") pod \"22d396fa-2f66-4e65-a81d-77a0818880a8\" (UID: \"22d396fa-2f66-4e65-a81d-77a0818880a8\") " Sep 10 15:06:31.745025 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.744965 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/754c410c-c882-490a-ba00-72ab36a5af9f-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.745025 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.744977 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh5ng\" (UniqueName: \"kubernetes.io/projected/754c410c-c882-490a-ba00-72ab36a5af9f-kube-api-access-hh5ng\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.745148 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.745120 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22d396fa-2f66-4e65-a81d-77a0818880a8" (UID: "22d396fa-2f66-4e65-a81d-77a0818880a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:31.747004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.746983 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm" (OuterVolumeSpecName: "kube-api-access-87whm") pod "22d396fa-2f66-4e65-a81d-77a0818880a8" (UID: "22d396fa-2f66-4e65-a81d-77a0818880a8"). InnerVolumeSpecName "kube-api-access-87whm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:31.773532 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.773498 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:31.845989 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.845953 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d396fa-2f66-4e65-a81d-77a0818880a8-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.845989 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:31.845991 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87whm\" (UniqueName: \"kubernetes.io/projected/22d396fa-2f66-4e65-a81d-77a0818880a8-kube-api-access-87whm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:06:32.100773 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.100662 2581 generic.go:356] "Generic (PLEG): container finished" podID="754c410c-c882-490a-ba00-72ab36a5af9f" containerID="85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2" exitCode=0 Sep 10 15:06:32.100773 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.100706 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" Sep 10 15:06:32.100773 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.100742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" event={"ID":"754c410c-c882-490a-ba00-72ab36a5af9f","Type":"ContainerDied","Data":"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2"} Sep 10 15:06:32.101054 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.100807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8" event={"ID":"754c410c-c882-490a-ba00-72ab36a5af9f","Type":"ContainerDied","Data":"a0a9e884e1aad99c6fca3ce6fd4cab6848163065cdb36d3471dbf586529c41eb"} Sep 10 15:06:32.101054 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.100831 2581 scope.go:117] "RemoveContainer" containerID="85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2" Sep 10 15:06:32.102521 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.102463 2581 generic.go:356] "Generic (PLEG): container finished" podID="22d396fa-2f66-4e65-a81d-77a0818880a8" containerID="bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70" exitCode=0 Sep 10 15:06:32.102618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.102557 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" event={"ID":"22d396fa-2f66-4e65-a81d-77a0818880a8","Type":"ContainerDied","Data":"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70"} Sep 10 15:06:32.102618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.102596 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" event={"ID":"22d396fa-2f66-4e65-a81d-77a0818880a8","Type":"ContainerDied","Data":"9904416a52c8ad5ae1ae8b08eca16099f83597eb70bbb0c9f525b0a3bdf01dd9"} Sep 10 15:06:32.102728 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.102643 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w" Sep 10 15:06:32.112965 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.112855 2581 scope.go:117] "RemoveContainer" containerID="85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2" Sep 10 15:06:32.113220 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:06:32.113199 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2\": container with ID starting with 85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2 not found: ID does not exist" containerID="85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2" Sep 10 15:06:32.113299 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.113227 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2"} err="failed to get container status \"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2\": rpc error: code = NotFound desc = could not find container \"85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2\": container with ID starting with 85e80f57c39cd00d3021317cad6c132d561dad78d13b75a81673bcbfa9f872d2 not found: ID does not exist" Sep 10 15:06:32.113299 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.113245 2581 scope.go:117] "RemoveContainer" containerID="bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70" Sep 10 15:06:32.125020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.124988 2581 scope.go:117] "RemoveContainer" containerID="bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70" Sep 10 15:06:32.125374 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:06:32.125352 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70\": container with ID starting with bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70 not found: ID does not exist" containerID="bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70" Sep 10 15:06:32.125468 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.125393 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70"} err="failed to get container status \"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70\": rpc error: code = NotFound desc = could not find container \"bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70\": container with ID starting with bb5105fc683732ecb24d21f988503ab3c3bea50b516a3e339088052f2324fc70 not found: ID does not exist" Sep 10 15:06:32.135482 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.135452 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:06:32.140021 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.139996 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-411a8086-545fdcb66f-ccbz8"] Sep 10 15:06:32.152882 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.152851 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:06:32.156596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.156572 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-411a8086-7555b6db5d-7826w"] Sep 10 15:06:32.237015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.236973 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d396fa-2f66-4e65-a81d-77a0818880a8" path="/var/lib/kubelet/pods/22d396fa-2f66-4e65-a81d-77a0818880a8/volumes" Sep 10 15:06:32.237306 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.237292 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d43821a-9ec0-4bde-9d50-e69f12cceabd" path="/var/lib/kubelet/pods/6d43821a-9ec0-4bde-9d50-e69f12cceabd/volumes" Sep 10 15:06:32.237598 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:32.237585 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754c410c-c882-490a-ba00-72ab36a5af9f" path="/var/lib/kubelet/pods/754c410c-c882-490a-ba00-72ab36a5af9f/volumes" Sep 10 15:06:38.330396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:38.330363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:38.438489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:38.438446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:38.632713 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:38.632618 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:38.995444 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:38.995409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:39.109810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:39.109751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:39.277647 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:39.277549 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:41.094941 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:41.094900 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:06:41.095437 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:41.095079 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:06:45.413636 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:45.413603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:45.530089 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:45.530053 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:45.638447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:45.638408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:45.829937 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:45.829908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:51.738084 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:51.737984 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:51.819284 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:51.819246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:52.110211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:52.110110 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:52.420491 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:52.420400 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:52.642657 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:52.642618 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:52.746268 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:52.746234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:52.971591 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:52.971555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:53.216849 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:53.216697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:53.621661 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:53.621619 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:56.397910 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:56.397868 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:56.648607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:56.648517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:56.779433 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:56.779381 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:57.701800 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:57.701741 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:57.778401 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:57.778366 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:58.067419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.067385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:58.242611 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.242577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:58.341143 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.341048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:58.382035 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.381998 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:06:58.382489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382472 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d396fa-2f66-4e65-a81d-77a0818880a8" containerName="payload-pre-processing" Sep 10 15:06:58.382489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382490 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d396fa-2f66-4e65-a81d-77a0818880a8" containerName="payload-pre-processing" Sep 10 15:06:58.382663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382500 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754c410c-c882-490a-ba00-72ab36a5af9f" containerName="payload-processing" Sep 10 15:06:58.382663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382510 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="754c410c-c882-490a-ba00-72ab36a5af9f" containerName="payload-processing" Sep 10 15:06:58.382663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382592 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="754c410c-c882-490a-ba00-72ab36a5af9f" containerName="payload-processing" Sep 10 15:06:58.382663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.382603 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d396fa-2f66-4e65-a81d-77a0818880a8" containerName="payload-pre-processing" Sep 10 15:06:58.385868 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.385848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.388146 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.388124 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d5a2df-dockercfg-mk7fp\"" Sep 10 15:06:58.389715 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.389694 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d5a2df\"" Sep 10 15:06:58.391865 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.391842 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:06:58.395389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.395366 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.404091 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.403981 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:06:58.408928 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.408902 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:06:58.484433 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.484402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lm7r\" (UniqueName: \"kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.484433 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.484438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.484660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.484461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.484660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.484593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj74\" (UniqueName: \"kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.585079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lm7r\" (UniqueName: \"kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.585079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.585343 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.585343 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj74\" (UniqueName: \"kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.585890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.586001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.585871 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.594778 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.594670 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lm7r\" (UniqueName: \"kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r\") pod \"payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.594914 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.594793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj74\" (UniqueName: \"kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74\") pod \"payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.696877 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.696829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:58.708568 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.708540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:58.747657 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.747605 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:58.852833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:58.852722 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:06:59.001911 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.001875 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:06:59.006449 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:59.006421 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5b0cca_6c3c_49f5_a9c9_510470ba29aa.slice/crio-714fd80aef52f2edb33ca223b7481aba77ecc507e22f3ed13a9dc17599af709f WatchSource:0}: Error finding container 714fd80aef52f2edb33ca223b7481aba77ecc507e22f3ed13a9dc17599af709f: Status 404 returned error can't find the container with id 714fd80aef52f2edb33ca223b7481aba77ecc507e22f3ed13a9dc17599af709f Sep 10 15:06:59.037702 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.037676 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:06:59.038180 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:06:59.038153 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033b206f_18ee_4703_bc44_266e9ce196d9.slice/crio-f027f82f3bca26f378a402ddc94ed534b1f2c2a67c154b996fd2d2314ec35dbc WatchSource:0}: Error finding container f027f82f3bca26f378a402ddc94ed534b1f2c2a67c154b996fd2d2314ec35dbc: Status 404 returned error can't find the container with id f027f82f3bca26f378a402ddc94ed534b1f2c2a67c154b996fd2d2314ec35dbc Sep 10 15:06:59.260961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.260918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" event={"ID":"033b206f-18ee-4703-bc44-266e9ce196d9","Type":"ContainerStarted","Data":"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5"} Sep 10 15:06:59.261172 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.261010 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" event={"ID":"033b206f-18ee-4703-bc44-266e9ce196d9","Type":"ContainerStarted","Data":"f027f82f3bca26f378a402ddc94ed534b1f2c2a67c154b996fd2d2314ec35dbc"} Sep 10 15:06:59.261172 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.261090 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:06:59.262712 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.262672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" event={"ID":"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa","Type":"ContainerStarted","Data":"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0"} Sep 10 15:06:59.262879 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.262740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:06:59.262879 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.262778 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" event={"ID":"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa","Type":"ContainerStarted","Data":"714fd80aef52f2edb33ca223b7481aba77ecc507e22f3ed13a9dc17599af709f"} Sep 10 15:06:59.280657 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.280598 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" podStartSLOduration=1.280582489 podStartE2EDuration="1.280582489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:59.279507581 +0000 UTC m=+1173.573809711" watchObservedRunningTime="2026-09-10 15:06:59.280582489 +0000 UTC m=+1173.574884617" Sep 10 15:06:59.305378 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.305316 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" podStartSLOduration=1.305299105 podStartE2EDuration="1.305299105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:59.302537514 +0000 UTC m=+1173.596839645" watchObservedRunningTime="2026-09-10 15:06:59.305299105 +0000 UTC m=+1173.599601234" Sep 10 15:06:59.526094 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.526011 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:06:59.529315 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.529291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.531702 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.531679 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0a9de4-dockercfg-jj45l\"" Sep 10 15:06:59.532578 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.532561 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0a9de4\"" Sep 10 15:06:59.542927 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.542903 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:06:59.546860 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.546840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.564553 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.564521 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:06:59.585498 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.585443 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:06:59.594877 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.594823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c92q\" (UniqueName: \"kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.595080 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.594920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.595080 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.594962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrk9\" (UniqueName: \"kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.595080 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.594996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.696097 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.696050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c92q\" (UniqueName: \"kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.696097 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.696103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.696365 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.696144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrk9\" (UniqueName: \"kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.696365 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.696168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.697010 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.696988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.697113 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.697023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.710663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.710631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrk9\" (UniqueName: \"kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9\") pod \"payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:06:59.711033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.710854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c92q\" (UniqueName: \"kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q\") pod \"payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.841027 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.840924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:06:59.857450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:06:59.857420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:07:00.124934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.124858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:07:00.128453 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:07:00.128423 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abd227d_6f49_4951_af35_a5eae7bf925c.slice/crio-99db4b94f9f90b1f4bf738566352165ed189733027f9f297ac3fa389657c1012 WatchSource:0}: Error finding container 99db4b94f9f90b1f4bf738566352165ed189733027f9f297ac3fa389657c1012: Status 404 returned error can't find the container with id 99db4b94f9f90b1f4bf738566352165ed189733027f9f297ac3fa389657c1012 Sep 10 15:07:00.157215 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.157192 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:07:00.163046 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:07:00.163019 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a02835_a6a2_4a5c_b058_74c146185ce1.slice/crio-06b9b07810d76c5d2c978588126b8c2a1eae8140c3634a2ce491caa7b071ab0e WatchSource:0}: Error finding container 06b9b07810d76c5d2c978588126b8c2a1eae8140c3634a2ce491caa7b071ab0e: Status 404 returned error can't find the container with id 06b9b07810d76c5d2c978588126b8c2a1eae8140c3634a2ce491caa7b071ab0e Sep 10 15:07:00.272159 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.272122 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" event={"ID":"d7a02835-a6a2-4a5c-b058-74c146185ce1","Type":"ContainerStarted","Data":"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad"} Sep 10 15:07:00.272374 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.272184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" event={"ID":"d7a02835-a6a2-4a5c-b058-74c146185ce1","Type":"ContainerStarted","Data":"06b9b07810d76c5d2c978588126b8c2a1eae8140c3634a2ce491caa7b071ab0e"} Sep 10 15:07:00.272374 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.272250 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:07:00.274162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.274107 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" event={"ID":"3abd227d-6f49-4951-af35-a5eae7bf925c","Type":"ContainerStarted","Data":"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a"} Sep 10 15:07:00.274300 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.274170 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:07:00.274300 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.274194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" event={"ID":"3abd227d-6f49-4951-af35-a5eae7bf925c","Type":"ContainerStarted","Data":"99db4b94f9f90b1f4bf738566352165ed189733027f9f297ac3fa389657c1012"} Sep 10 15:07:00.300086 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.300008 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" podStartSLOduration=1.2999855249999999 podStartE2EDuration="1.299985525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:00.296898548 +0000 UTC m=+1174.591200678" watchObservedRunningTime="2026-09-10 15:07:00.299985525 +0000 UTC m=+1174.594287656" Sep 10 15:07:00.320622 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.320551 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" podStartSLOduration=1.320527893 podStartE2EDuration="1.320527893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:00.317554704 +0000 UTC m=+1174.611856846" watchObservedRunningTime="2026-09-10 15:07:00.320527893 +0000 UTC m=+1174.614830025" Sep 10 15:07:00.403362 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.403256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:00.523312 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:00.523269 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:02.024143 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:02.024101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:10.276665 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:10.276633 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:07:10.277144 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:10.276712 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:07:11.283617 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:11.283587 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:07:11.284031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:11.283660 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:07:17.703713 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:17.703675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:17.825633 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:17.825580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:18.764354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:18.764317 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:19.963598 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:19.963561 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:20.712804 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:20.712744 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:21.296861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:21.296807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:21.403501 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:21.403470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:21.511392 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:21.511357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:21.635861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:21.635779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:21.967967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:21.967880 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:22.150742 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:22.150706 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:24.535143 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:24.535102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:24.653710 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:24.653664 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:25.698815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:25.698771 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:25.767007 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:25.766968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:26.000930 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.000891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:26.191740 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.191703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:26.235978 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.235942 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:07:26.238613 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.238590 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:07:26.292446 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.292361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:26.299450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.299423 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:07:26.303456 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.303434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.306243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.306214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1f985c\"" Sep 10 15:07:26.306243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.306224 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1f985c-dockercfg-w6t9w\"" Sep 10 15:07:26.314107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.314070 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:07:26.327530 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.327495 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:07:26.330901 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.330881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.343158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.343131 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:07:26.443096 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.443065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.443274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.443117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfc4\" (UniqueName: \"kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.443274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.443145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.443274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.443172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6vq\" (UniqueName: \"kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.544016 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.543909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.544016 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.543969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfc4\" (UniqueName: \"kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.544016 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.544002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.544316 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.544031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6vq\" (UniqueName: \"kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.544671 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.544648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.544709 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.544655 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.553265 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.553235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfc4\" (UniqueName: \"kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4\") pod \"payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.553419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.553271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6vq\" (UniqueName: \"kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq\") pod \"payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.615122 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.615090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:26.642484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.642452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:26.895521 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.895473 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:07:26.897903 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:07:26.897877 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5e50af_c2af_49bd_816b_3eeadec07c10.slice/crio-ed301aa73f0d08d614ea2f95e13e161b8f62bc2d5cd2d57faa9fcd9f71dc40ce WatchSource:0}: Error finding container ed301aa73f0d08d614ea2f95e13e161b8f62bc2d5cd2d57faa9fcd9f71dc40ce: Status 404 returned error can't find the container with id ed301aa73f0d08d614ea2f95e13e161b8f62bc2d5cd2d57faa9fcd9f71dc40ce Sep 10 15:07:26.933655 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:26.933583 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:07:26.936362 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:07:26.936332 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cda971e_0791_4d52_a15b_d95e2ef2648d.slice/crio-f97554a5edfba8a4768d651607e43db0e2a80a43798da226d03974ce1e754358 WatchSource:0}: Error finding container f97554a5edfba8a4768d651607e43db0e2a80a43798da226d03974ce1e754358: Status 404 returned error can't find the container with id f97554a5edfba8a4768d651607e43db0e2a80a43798da226d03974ce1e754358 Sep 10 15:07:27.435416 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.435375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" event={"ID":"6b5e50af-c2af-49bd-816b-3eeadec07c10","Type":"ContainerStarted","Data":"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737"} Sep 10 15:07:27.435643 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.435452 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:27.435643 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.435476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" event={"ID":"6b5e50af-c2af-49bd-816b-3eeadec07c10","Type":"ContainerStarted","Data":"ed301aa73f0d08d614ea2f95e13e161b8f62bc2d5cd2d57faa9fcd9f71dc40ce"} Sep 10 15:07:27.437345 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.437319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" event={"ID":"5cda971e-0791-4d52-a15b-d95e2ef2648d","Type":"ContainerStarted","Data":"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210"} Sep 10 15:07:27.437485 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.437371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" event={"ID":"5cda971e-0791-4d52-a15b-d95e2ef2648d","Type":"ContainerStarted","Data":"f97554a5edfba8a4768d651607e43db0e2a80a43798da226d03974ce1e754358"} Sep 10 15:07:27.437544 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.437508 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:27.453151 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.453094 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" podStartSLOduration=1.453076356 podStartE2EDuration="1.453076356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:27.451444783 +0000 UTC m=+1201.745746913" watchObservedRunningTime="2026-09-10 15:07:27.453076356 +0000 UTC m=+1201.747378487" Sep 10 15:07:27.467834 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:27.467747 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" podStartSLOduration=1.467718865 podStartE2EDuration="1.467718865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:27.467324716 +0000 UTC m=+1201.761626869" watchObservedRunningTime="2026-09-10 15:07:27.467718865 +0000 UTC m=+1201.762020995" Sep 10 15:07:31.567768 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:31.567711 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:38.446507 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:38.446471 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:07:38.448821 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:38.446939 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:07:43.086551 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:43.086508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:49.501268 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:49.501228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:58.213070 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:58.213040 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:58.548492 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:58.548453 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:07:58.897379 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:07:58.897290 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:01.579947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:01.579911 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:01.740671 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:01.740637 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:01.846151 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:01.846054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:02.900314 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:02.900277 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:02.966727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:02.966689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:03.426924 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.426887 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:03.580721 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.580688 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:08:03.584497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.584478 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.587806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.587775 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5c801c28\"" Sep 10 15:08:03.605217 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.605175 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:03.614052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.614017 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:08:03.670644 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.670614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x25m\" (UniqueName: \"kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.670849 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.670676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.740931 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.740888 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:08:03.744587 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.744566 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.749168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.749136 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5c801c28-dockercfg-vt749\"" Sep 10 15:08:03.764037 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.764005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:08:03.772077 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.772038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x25m\" (UniqueName: \"kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.772243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.772184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.773092 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.773066 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.789224 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.789195 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x25m\" (UniqueName: \"kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m\") pod \"payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.873059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.873014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7rc\" (UniqueName: \"kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.873253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.873101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.897645 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.897607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:03.974450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.974412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7rc\" (UniqueName: \"kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.974786 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.974498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.975262 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.975240 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:03.984806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:03.984779 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7rc\" (UniqueName: \"kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc\") pod \"payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:04.055682 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.055583 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:04.177546 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.177473 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:08:04.181549 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:08:04.181522 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd938d3aa_608f_4e73_ae02_e2fca4f7bcf1.slice/crio-fd8e365718760a3c6b665cf7aea6c1919643afb69bcd32127edd8afd3dab3949 WatchSource:0}: Error finding container fd8e365718760a3c6b665cf7aea6c1919643afb69bcd32127edd8afd3dab3949: Status 404 returned error can't find the container with id fd8e365718760a3c6b665cf7aea6c1919643afb69bcd32127edd8afd3dab3949 Sep 10 15:08:04.338512 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.338483 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:08:04.340006 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:08:04.339979 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02304173_ae77_4491_81ae_c1a21a232482.slice/crio-54f09d082b382e529a8f9b0a5b3ab59c5a82d7e8ff966c26c8bb8628be02353e WatchSource:0}: Error finding container 54f09d082b382e529a8f9b0a5b3ab59c5a82d7e8ff966c26c8bb8628be02353e: Status 404 returned error can't find the container with id 54f09d082b382e529a8f9b0a5b3ab59c5a82d7e8ff966c26c8bb8628be02353e Sep 10 15:08:04.657064 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.656965 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" event={"ID":"02304173-ae77-4491-81ae-c1a21a232482","Type":"ContainerStarted","Data":"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56"} Sep 10 15:08:04.657064 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.657034 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:04.657064 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.657062 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" event={"ID":"02304173-ae77-4491-81ae-c1a21a232482","Type":"ContainerStarted","Data":"54f09d082b382e529a8f9b0a5b3ab59c5a82d7e8ff966c26c8bb8628be02353e"} Sep 10 15:08:04.658897 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.658870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" event={"ID":"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1","Type":"ContainerStarted","Data":"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab"} Sep 10 15:08:04.659038 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.658914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" event={"ID":"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1","Type":"ContainerStarted","Data":"fd8e365718760a3c6b665cf7aea6c1919643afb69bcd32127edd8afd3dab3949"} Sep 10 15:08:04.659038 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.658943 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:04.676840 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.676786 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" podStartSLOduration=1.67676917 podStartE2EDuration="1.67676917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:04.675353194 +0000 UTC m=+1238.969655324" watchObservedRunningTime="2026-09-10 15:08:04.67676917 +0000 UTC m=+1238.971071293" Sep 10 15:08:04.702164 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:04.702091 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" podStartSLOduration=1.702073968 podStartE2EDuration="1.702073968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:04.700480289 +0000 UTC m=+1238.994782420" watchObservedRunningTime="2026-09-10 15:08:04.702073968 +0000 UTC m=+1238.996376105" Sep 10 15:08:11.622896 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:11.622863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:11.692023 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:11.691989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:12.994544 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:12.994503 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:15.670931 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:15.670898 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:08:15.671418 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:15.671064 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:08:21.032137 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:21.032052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:23.982581 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:23.982541 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:26.403469 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:26.403429 2581 scope.go:117] "RemoveContainer" containerID="93dc7e51c93208377eaba56be608dbbce7befc0c4077dd806f451c413ab71a0d" Sep 10 15:08:26.664977 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:26.664878 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:28.295995 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:28.295951 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:08:41.883836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:08:41.883797 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:11.022615 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.022578 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27"] Sep 10 15:09:11.026397 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.026381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:11.028997 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.028977 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:09:11.033623 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.033562 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27"] Sep 10 15:09:11.107894 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.107850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5c801c28-8wp27\" (UID: \"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:11.208822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.208783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5c801c28-8wp27\" (UID: \"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:11.209444 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.209420 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5c801c28-8wp27\" (UID: \"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:11.338869 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.338780 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:11.612107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:11.612070 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27"] Sep 10 15:09:11.614503 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:09:11.614473 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adbd7af_4bf0_48eb_b9d6_4080e17efb6d.slice/crio-619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c WatchSource:0}: Error finding container 619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c: Status 404 returned error can't find the container with id 619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c Sep 10 15:09:12.067954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:12.067914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" event={"ID":"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d","Type":"ContainerStarted","Data":"9e30b8ef2448309c04581e93b3d8b9cb2160cd695f47b02ff325f63e3da7d28b"} Sep 10 15:09:12.068389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:12.067976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" event={"ID":"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d","Type":"ContainerStarted","Data":"619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c"} Sep 10 15:09:12.084952 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:12.084900 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" podStartSLOduration=1.084883904 podStartE2EDuration="1.084883904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:09:12.08271333 +0000 UTC m=+1306.377015460" watchObservedRunningTime="2026-09-10 15:09:12.084883904 +0000 UTC m=+1306.379186032" Sep 10 15:09:13.075530 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:13.075496 2581 generic.go:356] "Generic (PLEG): container finished" podID="9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" containerID="9e30b8ef2448309c04581e93b3d8b9cb2160cd695f47b02ff325f63e3da7d28b" exitCode=0 Sep 10 15:09:13.075954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:13.075547 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" event={"ID":"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d","Type":"ContainerDied","Data":"9e30b8ef2448309c04581e93b3d8b9cb2160cd695f47b02ff325f63e3da7d28b"} Sep 10 15:09:14.225606 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:14.225575 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:14.336913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:14.336822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca\") pod \"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d\" (UID: \"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d\") " Sep 10 15:09:14.337255 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:14.337233 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" (UID: "9adbd7af-4bf0-48eb-b9d6-4080e17efb6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:09:14.438123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:14.438091 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:09:15.087658 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:15.087626 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" Sep 10 15:09:15.087658 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:15.087641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27" event={"ID":"9adbd7af-4bf0-48eb-b9d6-4080e17efb6d","Type":"ContainerDied","Data":"619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c"} Sep 10 15:09:15.087951 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:15.087683 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619549a4c617134b7ff9a9bb02007d9cc9b29e0ecd5a29a6f073845fcf68f56c" Sep 10 15:09:26.426147 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.426116 2581 scope.go:117] "RemoveContainer" containerID="c98502372b8375baea722a742ee511b8bc45df0a55c69ca8c3208eb97f1657dd" Sep 10 15:09:26.435209 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.435186 2581 scope.go:117] "RemoveContainer" containerID="a276a396a5dbcc0d1530fd00bd1d6cf120ddcd7dd48380cda318a50c8a6e845f" Sep 10 15:09:26.443726 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.443696 2581 scope.go:117] "RemoveContainer" containerID="efe220fbb02492f4a86f12557ca5581e946487257017c9a37bd458a36126367d" Sep 10 15:09:26.682503 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.682428 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:26.711510 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.711477 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:09:26.711768 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.711728 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" podUID="02304173-ae77-4491-81ae-c1a21a232482" containerName="payload-processing" containerID="cri-o://e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56" gracePeriod=30 Sep 10 15:09:26.718021 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.717984 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:09:26.718236 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.718209 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" podUID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" containerName="payload-pre-processing" containerID="cri-o://6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab" gracePeriod=30 Sep 10 15:09:26.757004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.756965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:26.838085 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.838050 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:26.883348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.883315 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27"] Sep 10 15:09:26.886499 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.886406 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5c801c28-8wp27"] Sep 10 15:09:26.910028 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:26.909972 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:27.011387 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.011361 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:09:27.014272 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.014250 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:09:27.152400 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.152364 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume\") pod \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " Sep 10 15:09:27.152604 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.152475 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz7rc\" (UniqueName: \"kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc\") pod \"02304173-ae77-4491-81ae-c1a21a232482\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " Sep 10 15:09:27.152604 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.152501 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x25m\" (UniqueName: \"kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m\") pod \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\" (UID: \"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1\") " Sep 10 15:09:27.152604 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.152551 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume\") pod \"02304173-ae77-4491-81ae-c1a21a232482\" (UID: \"02304173-ae77-4491-81ae-c1a21a232482\") " Sep 10 15:09:27.152908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.152874 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" (UID: "d938d3aa-608f-4e73-ae02-e2fca4f7bcf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:09:27.153100 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.153075 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02304173-ae77-4491-81ae-c1a21a232482" (UID: "02304173-ae77-4491-81ae-c1a21a232482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:09:27.154721 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.154694 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc" (OuterVolumeSpecName: "kube-api-access-fz7rc") pod "02304173-ae77-4491-81ae-c1a21a232482" (UID: "02304173-ae77-4491-81ae-c1a21a232482"). InnerVolumeSpecName "kube-api-access-fz7rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:09:27.154848 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.154794 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m" (OuterVolumeSpecName: "kube-api-access-8x25m") pod "d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" (UID: "d938d3aa-608f-4e73-ae02-e2fca4f7bcf1"). InnerVolumeSpecName "kube-api-access-8x25m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:09:27.158798 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.158770 2581 generic.go:356] "Generic (PLEG): container finished" podID="02304173-ae77-4491-81ae-c1a21a232482" containerID="e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56" exitCode=0 Sep 10 15:09:27.158893 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.158821 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" Sep 10 15:09:27.158893 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.158866 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" event={"ID":"02304173-ae77-4491-81ae-c1a21a232482","Type":"ContainerDied","Data":"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56"} Sep 10 15:09:27.158984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.158924 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t" event={"ID":"02304173-ae77-4491-81ae-c1a21a232482","Type":"ContainerDied","Data":"54f09d082b382e529a8f9b0a5b3ab59c5a82d7e8ff966c26c8bb8628be02353e"} Sep 10 15:09:27.158984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.158962 2581 scope.go:117] "RemoveContainer" containerID="e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56" Sep 10 15:09:27.160196 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.160175 2581 generic.go:356] "Generic (PLEG): container finished" podID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" containerID="6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab" exitCode=0 Sep 10 15:09:27.160271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.160215 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" Sep 10 15:09:27.160319 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.160264 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" event={"ID":"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1","Type":"ContainerDied","Data":"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab"} Sep 10 15:09:27.160377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.160324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6" event={"ID":"d938d3aa-608f-4e73-ae02-e2fca4f7bcf1","Type":"ContainerDied","Data":"fd8e365718760a3c6b665cf7aea6c1919643afb69bcd32127edd8afd3dab3949"} Sep 10 15:09:27.176605 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.176587 2581 scope.go:117] "RemoveContainer" containerID="e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56" Sep 10 15:09:27.176962 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:09:27.176940 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56\": container with ID starting with e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56 not found: ID does not exist" containerID="e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56" Sep 10 15:09:27.177023 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.176971 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56"} err="failed to get container status \"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56\": rpc error: code = NotFound desc = could not find container \"e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56\": container with ID starting with e405a70a672fe5c184adf055c10470d30d0aadbf7ac995d4c9bbe3fe47948f56 not found: ID does not exist" Sep 10 15:09:27.177023 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.176989 2581 scope.go:117] "RemoveContainer" containerID="6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab" Sep 10 15:09:27.186147 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.186126 2581 scope.go:117] "RemoveContainer" containerID="6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab" Sep 10 15:09:27.186219 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.186168 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:09:27.186517 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:09:27.186484 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab\": container with ID starting with 6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab not found: ID does not exist" containerID="6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab" Sep 10 15:09:27.186685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.186523 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab"} err="failed to get container status \"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab\": rpc error: code = NotFound desc = could not find container \"6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab\": container with ID starting with 6960df6833d419a98fd286f0498b840b450d2b58c3af7064750a64ea7bd0d7ab not found: ID does not exist" Sep 10 15:09:27.189026 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.189001 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5c801c28-6ddd5fbdbf-m7c9t"] Sep 10 15:09:27.199461 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.199432 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:09:27.205421 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.205398 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5c801c28-69cf5dc986-dwpd6"] Sep 10 15:09:27.253650 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.253620 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7rc\" (UniqueName: \"kubernetes.io/projected/02304173-ae77-4491-81ae-c1a21a232482-kube-api-access-fz7rc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:09:27.253837 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.253653 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x25m\" (UniqueName: \"kubernetes.io/projected/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-kube-api-access-8x25m\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:09:27.253837 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.253667 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02304173-ae77-4491-81ae-c1a21a232482-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:09:27.253837 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:27.253676 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:09:28.230409 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:28.230366 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02304173-ae77-4491-81ae-c1a21a232482" path="/var/lib/kubelet/pods/02304173-ae77-4491-81ae-c1a21a232482/volumes" Sep 10 15:09:28.230935 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:28.230866 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" path="/var/lib/kubelet/pods/9adbd7af-4bf0-48eb-b9d6-4080e17efb6d/volumes" Sep 10 15:09:28.231255 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:28.231235 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" path="/var/lib/kubelet/pods/d938d3aa-608f-4e73-ae02-e2fca4f7bcf1/volumes" Sep 10 15:09:30.006823 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:30.006794 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:09:35.014320 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:09:35.014270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:06.701161 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:06.701125 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:07.754160 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:07.754122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:12.050624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:12.050593 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:12.169228 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:12.169194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:17.769151 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:17.769115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:19.724820 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:19.724785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:23.858601 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:23.858565 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:23.965982 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:23.965952 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:24.126364 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:24.126279 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:25.347962 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:25.347930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:25.447861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:25.447826 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:25.566381 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:25.566343 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:26.228099 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:26.228052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:26.370221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:26.370178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:26.471662 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:26.471630 2581 scope.go:117] "RemoveContainer" containerID="3c9fed00efe83fc7aa92c1c7a546674ecd03c64b7477f378fb4f29acefd7902d" Sep 10 15:10:27.615603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:27.615568 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:29.974053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:29.974014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:33.531265 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:33.531233 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:33.599043 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:33.599006 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:34.180933 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:34.180899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:40.090610 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:40.090572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:40.200683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:40.200648 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:40.296111 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:40.296080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:50.781365 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:50.781324 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:10:50.989708 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:10:50.989670 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:11.917335 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:11.917300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:17.797692 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:17.797604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:17.867879 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:17.867845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:18.150346 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:18.150244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:18.446214 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:18.446138 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:18.602357 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:18.602303 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:21.885104 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:21.885068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:21.971898 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:21.971858 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.218498 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.218467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.297497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.297461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.629693 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.629659 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.728773 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.728713 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.825460 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.825427 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:11:23.826004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.825977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:23.826004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.825993 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" containerName="revoke-keys" Sep 10 15:11:23.826004 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826010 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" containerName="revoke-keys" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826031 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" containerName="payload-pre-processing" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826039 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" containerName="payload-pre-processing" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826057 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02304173-ae77-4491-81ae-c1a21a232482" containerName="payload-processing" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826064 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02304173-ae77-4491-81ae-c1a21a232482" containerName="payload-processing" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826158 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d938d3aa-608f-4e73-ae02-e2fca4f7bcf1" containerName="payload-pre-processing" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826169 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adbd7af-4bf0-48eb-b9d6-4080e17efb6d" containerName="revoke-keys" Sep 10 15:11:23.826221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.826178 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02304173-ae77-4491-81ae-c1a21a232482" containerName="payload-processing" Sep 10 15:11:23.829250 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.829236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:23.832067 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.832050 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3e7a73\"" Sep 10 15:11:23.832169 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.832075 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3e7a73-dockercfg-gvftr\"" Sep 10 15:11:23.841547 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.841525 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:11:23.856939 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.856901 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:11:23.860474 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.860453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:23.870053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.870030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:11:23.897290 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.897192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87tk\" (UniqueName: \"kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:23.897457 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.897325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:23.998563 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.998522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87tk\" (UniqueName: \"kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:23.998788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.998612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:23.998788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.998639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:23.998788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.998664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvf4\" (UniqueName: \"kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:23.999352 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:23.999331 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:24.008263 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.008235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87tk\" (UniqueName: \"kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk\") pod \"payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:24.100003 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.099961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.100184 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.100014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvf4\" (UniqueName: \"kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.100813 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.100793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.110774 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.110738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvf4\" (UniqueName: \"kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4\") pod \"payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.139525 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.139491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:24.170798 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.170768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.416192 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.416166 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:11:24.418556 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:11:24.418525 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ff4f5_d5eb_433d_b99e_499e5991faae.slice/crio-f71ff91347eb22a18e81d115b5aceb89cd0214bf2537bfa52f2d71132b87fb6c WatchSource:0}: Error finding container f71ff91347eb22a18e81d115b5aceb89cd0214bf2537bfa52f2d71132b87fb6c: Status 404 returned error can't find the container with id f71ff91347eb22a18e81d115b5aceb89cd0214bf2537bfa52f2d71132b87fb6c Sep 10 15:11:24.463551 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.463526 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:11:24.473873 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:11:24.473845 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33539ea2_3747_4787_9e40_d3769737fc5e.slice/crio-2b0e1789221bf8f15e218a545f392dc1da1217d153bcbf4aadd3e65174ce93b4 WatchSource:0}: Error finding container 2b0e1789221bf8f15e218a545f392dc1da1217d153bcbf4aadd3e65174ce93b4: Status 404 returned error can't find the container with id 2b0e1789221bf8f15e218a545f392dc1da1217d153bcbf4aadd3e65174ce93b4 Sep 10 15:11:24.826277 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.826236 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" event={"ID":"33539ea2-3747-4787-9e40-d3769737fc5e","Type":"ContainerStarted","Data":"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f"} Sep 10 15:11:24.826458 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.826292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" event={"ID":"33539ea2-3747-4787-9e40-d3769737fc5e","Type":"ContainerStarted","Data":"2b0e1789221bf8f15e218a545f392dc1da1217d153bcbf4aadd3e65174ce93b4"} Sep 10 15:11:24.826458 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.826407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:24.837203 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.837163 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" event={"ID":"7e2ff4f5-d5eb-433d-b99e-499e5991faae","Type":"ContainerStarted","Data":"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0"} Sep 10 15:11:24.837371 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.837257 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" event={"ID":"7e2ff4f5-d5eb-433d-b99e-499e5991faae","Type":"ContainerStarted","Data":"f71ff91347eb22a18e81d115b5aceb89cd0214bf2537bfa52f2d71132b87fb6c"} Sep 10 15:11:24.837371 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.837287 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:24.846642 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.846590 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" podStartSLOduration=1.846574109 podStartE2EDuration="1.846574109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:24.844346802 +0000 UTC m=+1439.138648933" watchObservedRunningTime="2026-09-10 15:11:24.846574109 +0000 UTC m=+1439.140876242" Sep 10 15:11:24.861738 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:24.861685 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" podStartSLOduration=1.861667965 podStartE2EDuration="1.861667965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:24.860475806 +0000 UTC m=+1439.154777935" watchObservedRunningTime="2026-09-10 15:11:24.861667965 +0000 UTC m=+1439.155970093" Sep 10 15:11:26.504860 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:26.504827 2581 scope.go:117] "RemoveContainer" containerID="4a90846da7d23aed4d1c07a25b57b948d5a5ee9d580237010727dd633996cbdd" Sep 10 15:11:29.750873 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:29.750840 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:35.845451 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:35.845418 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:11:35.845871 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:35.845488 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:11:47.360020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:47.359977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:47.534794 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:47.534741 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:47.628719 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:47.628621 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:47.934321 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:47.934224 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:50.989351 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:50.989315 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:51.121347 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:51.121314 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:51.244690 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:51.244599 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.304525 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.304490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.412624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.412586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.502273 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.502240 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.801036 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.801008 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.903149 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.903111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:11:52.943495 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.943454 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:11:52.950267 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.950236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:52.953215 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.953191 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d7aa05-dockercfg-7f9rx\"" Sep 10 15:11:52.953579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.953561 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d7aa05\"" Sep 10 15:11:52.960465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:52.960438 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:11:53.000432 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.000395 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:11:53.004068 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.004043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.014401 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.014376 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:11:53.048180 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.048144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcpn\" (UniqueName: \"kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.048395 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.048205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.048395 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.048304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.048519 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.048399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cp4\" (UniqueName: \"kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.149372 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.149281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.149372 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.149341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.149610 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.149412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cp4\" (UniqueName: \"kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.149610 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.149454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcpn\" (UniqueName: \"kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.150116 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.150093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.150238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.150114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.158571 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.158543 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcpn\" (UniqueName: \"kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn\") pod \"payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.158571 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.158564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cp4\" (UniqueName: \"kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4\") pod \"payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.262213 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.262177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:53.315467 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.315436 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:53.542213 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.542185 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:11:53.543852 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:11:53.543824 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327464b9_e1f2_4119_bc05_4b88c24f1427.slice/crio-28afa3811dc83e6ab9382bed42e293674188f830be2736cb95a9ea4b5dbce657 WatchSource:0}: Error finding container 28afa3811dc83e6ab9382bed42e293674188f830be2736cb95a9ea4b5dbce657: Status 404 returned error can't find the container with id 28afa3811dc83e6ab9382bed42e293674188f830be2736cb95a9ea4b5dbce657 Sep 10 15:11:53.618425 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:53.618396 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:11:53.618727 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:11:53.618706 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8709a1c_1199_4240_95f2_209c738f707c.slice/crio-d24b3121f61723c3901fa3fae2cdf7ea7f4c4ff3de74d568cccd1678b6ca1005 WatchSource:0}: Error finding container d24b3121f61723c3901fa3fae2cdf7ea7f4c4ff3de74d568cccd1678b6ca1005: Status 404 returned error can't find the container with id d24b3121f61723c3901fa3fae2cdf7ea7f4c4ff3de74d568cccd1678b6ca1005 Sep 10 15:11:54.006323 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.006282 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" event={"ID":"d8709a1c-1199-4240-95f2-209c738f707c","Type":"ContainerStarted","Data":"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d"} Sep 10 15:11:54.006542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.006352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" event={"ID":"d8709a1c-1199-4240-95f2-209c738f707c","Type":"ContainerStarted","Data":"d24b3121f61723c3901fa3fae2cdf7ea7f4c4ff3de74d568cccd1678b6ca1005"} Sep 10 15:11:54.006542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.006384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:11:54.008328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.008299 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" event={"ID":"327464b9-e1f2-4119-bc05-4b88c24f1427","Type":"ContainerStarted","Data":"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f"} Sep 10 15:11:54.008473 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.008358 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:11:54.008473 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.008381 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" event={"ID":"327464b9-e1f2-4119-bc05-4b88c24f1427","Type":"ContainerStarted","Data":"28afa3811dc83e6ab9382bed42e293674188f830be2736cb95a9ea4b5dbce657"} Sep 10 15:11:54.022714 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.022657 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" podStartSLOduration=2.022639731 podStartE2EDuration="2.022639731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:54.02198058 +0000 UTC m=+1468.316282710" watchObservedRunningTime="2026-09-10 15:11:54.022639731 +0000 UTC m=+1468.316941861" Sep 10 15:11:54.038692 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:11:54.038649 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" podStartSLOduration=2.038636403 podStartE2EDuration="2.038636403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:54.037794349 +0000 UTC m=+1468.332096479" watchObservedRunningTime="2026-09-10 15:11:54.038636403 +0000 UTC m=+1468.332938532" Sep 10 15:12:05.017719 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:05.017680 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:12:05.018204 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:05.017795 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:12:15.907875 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:15.907835 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:18.754594 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.754557 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd"] Sep 10 15:12:18.758312 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.758295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:18.760818 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.760771 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:12:18.769150 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.769120 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd"] Sep 10 15:12:18.888490 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.888452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7a73-442zd\" (UID: \"d0474478-8e34-4f19-968d-679a29e4f811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:18.903651 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.903612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:18.989200 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.989161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7a73-442zd\" (UID: \"d0474478-8e34-4f19-968d-679a29e4f811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:18.989796 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:18.989776 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e7a73-442zd\" (UID: \"d0474478-8e34-4f19-968d-679a29e4f811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:19.077512 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:19.077430 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:19.348679 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:19.348652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd"] Sep 10 15:12:19.350907 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:12:19.350879 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0474478_8e34_4f19_968d_679a29e4f811.slice/crio-6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828 WatchSource:0}: Error finding container 6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828: Status 404 returned error can't find the container with id 6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828 Sep 10 15:12:19.799481 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:19.799450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:20.168806 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:20.168769 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" event={"ID":"d0474478-8e34-4f19-968d-679a29e4f811","Type":"ContainerStarted","Data":"3dc47ff6019337c63b57aaaee945c76db6a1dfc0c803679d503ca4160b9dab79"} Sep 10 15:12:20.168993 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:20.168821 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" event={"ID":"d0474478-8e34-4f19-968d-679a29e4f811","Type":"ContainerStarted","Data":"6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828"} Sep 10 15:12:20.204281 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:20.204233 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" podStartSLOduration=2.204214431 podStartE2EDuration="2.204214431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:12:20.202888246 +0000 UTC m=+1494.497190376" watchObservedRunningTime="2026-09-10 15:12:20.204214431 +0000 UTC m=+1494.498516560" Sep 10 15:12:20.556176 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:20.556135 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:20.663642 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:20.663616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:21.175421 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:21.175376 2581 generic.go:356] "Generic (PLEG): container finished" podID="d0474478-8e34-4f19-968d-679a29e4f811" containerID="3dc47ff6019337c63b57aaaee945c76db6a1dfc0c803679d503ca4160b9dab79" exitCode=0 Sep 10 15:12:21.175880 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:21.175462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" event={"ID":"d0474478-8e34-4f19-968d-679a29e4f811","Type":"ContainerDied","Data":"3dc47ff6019337c63b57aaaee945c76db6a1dfc0c803679d503ca4160b9dab79"} Sep 10 15:12:21.385542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:21.385510 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:22.332444 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:22.332420 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:22.421927 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:22.421890 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca\") pod \"d0474478-8e34-4f19-968d-679a29e4f811\" (UID: \"d0474478-8e34-4f19-968d-679a29e4f811\") " Sep 10 15:12:22.422289 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:22.422264 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0474478-8e34-4f19-968d-679a29e4f811" (UID: "d0474478-8e34-4f19-968d-679a29e4f811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:22.523611 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:22.523569 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0474478-8e34-4f19-968d-679a29e4f811-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:23.193947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:23.193915 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" Sep 10 15:12:23.194120 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:23.193935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd" event={"ID":"d0474478-8e34-4f19-968d-679a29e4f811","Type":"ContainerDied","Data":"6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828"} Sep 10 15:12:23.194120 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:23.194028 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd0b616c44262a8112231b5e179e857c7a48eed95f02f946d5899d0c967d828" Sep 10 15:12:26.275405 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:26.275374 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:12:26.278346 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:26.278321 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:12:26.528010 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:26.527923 2581 scope.go:117] "RemoveContainer" containerID="e1fc3254b137a508d98282b1d8b92b12b90d12ec8c5a9acbd961f5d6965b80b4" Sep 10 15:12:26.537687 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:26.537660 2581 scope.go:117] "RemoveContainer" containerID="67009177e524ddb00320c53bdaf13dd2c8abe0c614ee2d3fea670c0d2e5ebc9d" Sep 10 15:12:26.547044 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:26.547023 2581 scope.go:117] "RemoveContainer" containerID="b8cdaf75bc99b9ae91acd1dc64c5929c7a1b7d8106a0e8d55a78efa4a9d67968" Sep 10 15:12:28.900683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:28.900647 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:32.076620 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:32.076577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:33.068575 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:33.068538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:34.503560 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.503529 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:34.521958 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.521926 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:12:34.522197 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.522175 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" podUID="33539ea2-3747-4787-9e40-d3769737fc5e" containerName="payload-processing" containerID="cri-o://a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f" gracePeriod=30 Sep 10 15:12:34.537787 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.537739 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:12:34.538021 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.537983 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" podUID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" containerName="payload-pre-processing" containerID="cri-o://1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0" gracePeriod=30 Sep 10 15:12:34.609685 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.609652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:34.707253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.707189 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:34.744905 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.744873 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd"] Sep 10 15:12:34.748816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.748737 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e7a73-442zd"] Sep 10 15:12:34.799200 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.799172 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:12:34.806804 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.806779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:34.835285 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.835259 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:12:34.936377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936345 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g87tk\" (UniqueName: \"kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk\") pod \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " Sep 10 15:12:34.936377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936382 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume\") pod \"33539ea2-3747-4787-9e40-d3769737fc5e\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " Sep 10 15:12:34.936595 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936469 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume\") pod \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\" (UID: \"7e2ff4f5-d5eb-433d-b99e-499e5991faae\") " Sep 10 15:12:34.936595 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936486 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gvf4\" (UniqueName: \"kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4\") pod \"33539ea2-3747-4787-9e40-d3769737fc5e\" (UID: \"33539ea2-3747-4787-9e40-d3769737fc5e\") " Sep 10 15:12:34.936901 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936879 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33539ea2-3747-4787-9e40-d3769737fc5e" (UID: "33539ea2-3747-4787-9e40-d3769737fc5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:34.936988 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.936877 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e2ff4f5-d5eb-433d-b99e-499e5991faae" (UID: "7e2ff4f5-d5eb-433d-b99e-499e5991faae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:34.938885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.938857 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk" (OuterVolumeSpecName: "kube-api-access-g87tk") pod "7e2ff4f5-d5eb-433d-b99e-499e5991faae" (UID: "7e2ff4f5-d5eb-433d-b99e-499e5991faae"). InnerVolumeSpecName "kube-api-access-g87tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:34.938952 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:34.938863 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4" (OuterVolumeSpecName: "kube-api-access-9gvf4") pod "33539ea2-3747-4787-9e40-d3769737fc5e" (UID: "33539ea2-3747-4787-9e40-d3769737fc5e"). InnerVolumeSpecName "kube-api-access-9gvf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:35.038238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.038148 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g87tk\" (UniqueName: \"kubernetes.io/projected/7e2ff4f5-d5eb-433d-b99e-499e5991faae-kube-api-access-g87tk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:35.038238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.038180 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33539ea2-3747-4787-9e40-d3769737fc5e-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:35.038238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.038190 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ff4f5-d5eb-433d-b99e-499e5991faae-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:35.038238 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.038199 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvf4\" (UniqueName: \"kubernetes.io/projected/33539ea2-3747-4787-9e40-d3769737fc5e-kube-api-access-9gvf4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:35.104047 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.104008 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:35.264931 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.264896 2581 generic.go:356] "Generic (PLEG): container finished" podID="33539ea2-3747-4787-9e40-d3769737fc5e" containerID="a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f" exitCode=0 Sep 10 15:12:35.265133 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.264947 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" Sep 10 15:12:35.265133 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.264997 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" event={"ID":"33539ea2-3747-4787-9e40-d3769737fc5e","Type":"ContainerDied","Data":"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f"} Sep 10 15:12:35.265133 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.265059 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5" event={"ID":"33539ea2-3747-4787-9e40-d3769737fc5e","Type":"ContainerDied","Data":"2b0e1789221bf8f15e218a545f392dc1da1217d153bcbf4aadd3e65174ce93b4"} Sep 10 15:12:35.265133 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.265090 2581 scope.go:117] "RemoveContainer" containerID="a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f" Sep 10 15:12:35.266447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.266421 2581 generic.go:356] "Generic (PLEG): container finished" podID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" containerID="1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0" exitCode=0 Sep 10 15:12:35.266567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.266472 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" Sep 10 15:12:35.266567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.266494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" event={"ID":"7e2ff4f5-d5eb-433d-b99e-499e5991faae","Type":"ContainerDied","Data":"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0"} Sep 10 15:12:35.266567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.266553 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7" event={"ID":"7e2ff4f5-d5eb-433d-b99e-499e5991faae","Type":"ContainerDied","Data":"f71ff91347eb22a18e81d115b5aceb89cd0214bf2537bfa52f2d71132b87fb6c"} Sep 10 15:12:35.276280 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.276265 2581 scope.go:117] "RemoveContainer" containerID="a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f" Sep 10 15:12:35.276571 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:12:35.276546 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f\": container with ID starting with a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f not found: ID does not exist" containerID="a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f" Sep 10 15:12:35.276666 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.276585 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f"} err="failed to get container status \"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f\": rpc error: code = NotFound desc = could not find container \"a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f\": container with ID starting with a726b96fdd756b52c59db4032de8d0774fa78b348edd4baaaee03d0c9b30200f not found: ID does not exist" Sep 10 15:12:35.276666 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.276609 2581 scope.go:117] "RemoveContainer" containerID="1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0" Sep 10 15:12:35.285031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.285014 2581 scope.go:117] "RemoveContainer" containerID="1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0" Sep 10 15:12:35.285272 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:12:35.285255 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0\": container with ID starting with 1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0 not found: ID does not exist" containerID="1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0" Sep 10 15:12:35.285344 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.285284 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0"} err="failed to get container status \"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0\": rpc error: code = NotFound desc = could not find container \"1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0\": container with ID starting with 1ca8eb68f82dd789153e1b1df1a5a8f473fafb7a7c165a31e95e238b80b0a1e0 not found: ID does not exist" Sep 10 15:12:35.288978 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.288932 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:12:35.293113 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.293092 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7a73-5648b865b7-tzhl5"] Sep 10 15:12:35.304281 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.304259 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:12:35.310695 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:35.310670 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7a73-6fb874b794-glgm7"] Sep 10 15:12:36.230391 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:36.230360 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33539ea2-3747-4787-9e40-d3769737fc5e" path="/var/lib/kubelet/pods/33539ea2-3747-4787-9e40-d3769737fc5e/volumes" Sep 10 15:12:36.230808 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:36.230717 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" path="/var/lib/kubelet/pods/7e2ff4f5-d5eb-433d-b99e-499e5991faae/volumes" Sep 10 15:12:36.231029 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:36.231016 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0474478-8e34-4f19-968d-679a29e4f811" path="/var/lib/kubelet/pods/d0474478-8e34-4f19-968d-679a29e4f811/volumes" Sep 10 15:12:40.657746 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:40.657713 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:41.265379 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.265341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:41.365513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.365480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:41.540510 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.540424 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:41.836463 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836388 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m"] Sep 10 15:12:41.836856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836842 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33539ea2-3747-4787-9e40-d3769737fc5e" containerName="payload-processing" Sep 10 15:12:41.836915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836858 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33539ea2-3747-4787-9e40-d3769737fc5e" containerName="payload-processing" Sep 10 15:12:41.836915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836882 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" containerName="payload-pre-processing" Sep 10 15:12:41.836915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836887 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" containerName="payload-pre-processing" Sep 10 15:12:41.836915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836894 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0474478-8e34-4f19-968d-679a29e4f811" containerName="revoke-keys" Sep 10 15:12:41.836915 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836899 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0474478-8e34-4f19-968d-679a29e4f811" containerName="revoke-keys" Sep 10 15:12:41.837075 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836965 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0474478-8e34-4f19-968d-679a29e4f811" containerName="revoke-keys" Sep 10 15:12:41.837075 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836972 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33539ea2-3747-4787-9e40-d3769737fc5e" containerName="payload-processing" Sep 10 15:12:41.837075 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.836978 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2ff4f5-d5eb-433d-b99e-499e5991faae" containerName="payload-pre-processing" Sep 10 15:12:41.840001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.839985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:41.842316 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.842295 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:12:41.847718 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.847612 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m"] Sep 10 15:12:41.903996 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:41.903958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m\" (UID: \"0b4c627c-d991-4b4c-92ea-8ed56b630c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:42.004489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:42.004453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m\" (UID: \"0b4c627c-d991-4b4c-92ea-8ed56b630c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:42.005055 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:42.005035 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m\" (UID: \"0b4c627c-d991-4b4c-92ea-8ed56b630c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:42.097496 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:42.097408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:42.152280 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:42.152248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:42.426522 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:42.426497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m"] Sep 10 15:12:42.428584 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:12:42.428559 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4c627c_d991_4b4c_92ea_8ed56b630c72.slice/crio-86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07 WatchSource:0}: Error finding container 86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07: Status 404 returned error can't find the container with id 86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07 Sep 10 15:12:43.314480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:43.314443 2581 generic.go:356] "Generic (PLEG): container finished" podID="0b4c627c-d991-4b4c-92ea-8ed56b630c72" containerID="680799365e4aa7c9daeffd504ff971cba6c854aa5228cd5dae33d04b5d29773c" exitCode=0 Sep 10 15:12:43.314930 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:43.314485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" event={"ID":"0b4c627c-d991-4b4c-92ea-8ed56b630c72","Type":"ContainerDied","Data":"680799365e4aa7c9daeffd504ff971cba6c854aa5228cd5dae33d04b5d29773c"} Sep 10 15:12:43.314930 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:43.314540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" event={"ID":"0b4c627c-d991-4b4c-92ea-8ed56b630c72","Type":"ContainerStarted","Data":"86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07"} Sep 10 15:12:44.462863 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:44.462832 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:44.526039 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:44.526004 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca\") pod \"0b4c627c-d991-4b4c-92ea-8ed56b630c72\" (UID: \"0b4c627c-d991-4b4c-92ea-8ed56b630c72\") " Sep 10 15:12:44.526403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:44.526375 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b4c627c-d991-4b4c-92ea-8ed56b630c72" (UID: "0b4c627c-d991-4b4c-92ea-8ed56b630c72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:44.626790 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:44.626668 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4c627c-d991-4b4c-92ea-8ed56b630c72-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:45.327150 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:45.327117 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" Sep 10 15:12:45.327378 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:45.327148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m" event={"ID":"0b4c627c-d991-4b4c-92ea-8ed56b630c72","Type":"ContainerDied","Data":"86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07"} Sep 10 15:12:45.327378 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:45.327195 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86774984b9f7dc2ab6c501a95db063498514fd630c418ea64875900a9ae83c07" Sep 10 15:12:47.187605 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:47.187499 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:51.979139 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:51.979103 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:57.659689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.659654 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:57.681292 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.681253 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:12:57.681592 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.681557 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" podUID="327464b9-e1f2-4119-bc05-4b88c24f1427" containerName="payload-processing" containerID="cri-o://2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f" gracePeriod=30 Sep 10 15:12:57.687123 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.687095 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:12:57.687630 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.687597 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" podUID="d8709a1c-1199-4240-95f2-209c738f707c" containerName="payload-pre-processing" containerID="cri-o://133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d" gracePeriod=30 Sep 10 15:12:57.728070 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.727983 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:57.795466 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.795411 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:57.856561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.856525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m"] Sep 10 15:12:57.862482 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.862396 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7aa05-hmb7m"] Sep 10 15:12:57.866378 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.866346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:57.984987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.984961 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:12:57.987950 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:57.987925 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:12:58.046767 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.046721 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume\") pod \"327464b9-e1f2-4119-bc05-4b88c24f1427\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " Sep 10 15:12:58.046974 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.046781 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4cp4\" (UniqueName: \"kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4\") pod \"d8709a1c-1199-4240-95f2-209c738f707c\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " Sep 10 15:12:58.046974 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.046864 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hcpn\" (UniqueName: \"kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn\") pod \"327464b9-e1f2-4119-bc05-4b88c24f1427\" (UID: \"327464b9-e1f2-4119-bc05-4b88c24f1427\") " Sep 10 15:12:58.046974 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.046913 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume\") pod \"d8709a1c-1199-4240-95f2-209c738f707c\" (UID: \"d8709a1c-1199-4240-95f2-209c738f707c\") " Sep 10 15:12:58.047154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.047121 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "327464b9-e1f2-4119-bc05-4b88c24f1427" (UID: "327464b9-e1f2-4119-bc05-4b88c24f1427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:58.047335 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.047316 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8709a1c-1199-4240-95f2-209c738f707c" (UID: "d8709a1c-1199-4240-95f2-209c738f707c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:58.049076 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.049053 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4" (OuterVolumeSpecName: "kube-api-access-p4cp4") pod "d8709a1c-1199-4240-95f2-209c738f707c" (UID: "d8709a1c-1199-4240-95f2-209c738f707c"). InnerVolumeSpecName "kube-api-access-p4cp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:58.049201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.049176 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn" (OuterVolumeSpecName: "kube-api-access-5hcpn") pod "327464b9-e1f2-4119-bc05-4b88c24f1427" (UID: "327464b9-e1f2-4119-bc05-4b88c24f1427"). InnerVolumeSpecName "kube-api-access-5hcpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:58.148164 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.148126 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8709a1c-1199-4240-95f2-209c738f707c-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:58.148164 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.148157 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327464b9-e1f2-4119-bc05-4b88c24f1427-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:58.148164 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.148168 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cp4\" (UniqueName: \"kubernetes.io/projected/d8709a1c-1199-4240-95f2-209c738f707c-kube-api-access-p4cp4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:58.148414 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.148178 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hcpn\" (UniqueName: \"kubernetes.io/projected/327464b9-e1f2-4119-bc05-4b88c24f1427-kube-api-access-5hcpn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:12:58.209794 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.209662 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:12:58.230497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.230464 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b4c627c-d991-4b4c-92ea-8ed56b630c72" path="/var/lib/kubelet/pods/0b4c627c-d991-4b4c-92ea-8ed56b630c72/volumes" Sep 10 15:12:58.402158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.402122 2581 generic.go:356] "Generic (PLEG): container finished" podID="d8709a1c-1199-4240-95f2-209c738f707c" containerID="133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d" exitCode=0 Sep 10 15:12:58.402354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.402172 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" Sep 10 15:12:58.402354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.402217 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" event={"ID":"d8709a1c-1199-4240-95f2-209c738f707c","Type":"ContainerDied","Data":"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d"} Sep 10 15:12:58.402354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.402275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b" event={"ID":"d8709a1c-1199-4240-95f2-209c738f707c","Type":"ContainerDied","Data":"d24b3121f61723c3901fa3fae2cdf7ea7f4c4ff3de74d568cccd1678b6ca1005"} Sep 10 15:12:58.402354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.402304 2581 scope.go:117] "RemoveContainer" containerID="133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d" Sep 10 15:12:58.403731 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.403711 2581 generic.go:356] "Generic (PLEG): container finished" podID="327464b9-e1f2-4119-bc05-4b88c24f1427" containerID="2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f" exitCode=0 Sep 10 15:12:58.403872 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.403795 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" Sep 10 15:12:58.403872 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.403819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" event={"ID":"327464b9-e1f2-4119-bc05-4b88c24f1427","Type":"ContainerDied","Data":"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f"} Sep 10 15:12:58.403990 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.403885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6" event={"ID":"327464b9-e1f2-4119-bc05-4b88c24f1427","Type":"ContainerDied","Data":"28afa3811dc83e6ab9382bed42e293674188f830be2736cb95a9ea4b5dbce657"} Sep 10 15:12:58.412087 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.412071 2581 scope.go:117] "RemoveContainer" containerID="133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d" Sep 10 15:12:58.412363 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:12:58.412340 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d\": container with ID starting with 133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d not found: ID does not exist" containerID="133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d" Sep 10 15:12:58.412468 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.412369 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d"} err="failed to get container status \"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d\": rpc error: code = NotFound desc = could not find container \"133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d\": container with ID starting with 133efe6bed471bad72e7a7627bb878f3d97e14dda2ea93e72eb8cff07a2c4b0d not found: ID does not exist" Sep 10 15:12:58.412468 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.412386 2581 scope.go:117] "RemoveContainer" containerID="2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f" Sep 10 15:12:58.421961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.421942 2581 scope.go:117] "RemoveContainer" containerID="2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f" Sep 10 15:12:58.422251 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:12:58.422231 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f\": container with ID starting with 2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f not found: ID does not exist" containerID="2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f" Sep 10 15:12:58.422325 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.422260 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f"} err="failed to get container status \"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f\": rpc error: code = NotFound desc = could not find container \"2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f\": container with ID starting with 2a1d7d638212ecb58892769e304983a51239f22e7ad6e2a8425249c1000fd63f not found: ID does not exist" Sep 10 15:12:58.423949 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.423930 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:12:58.429739 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.429719 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d7aa05-748b7c77d5-2j7g6"] Sep 10 15:12:58.440793 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.440742 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:12:58.444898 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:12:58.444876 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7aa05-65f868bcdb-k4w5b"] Sep 10 15:13:00.230299 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:00.230261 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="327464b9-e1f2-4119-bc05-4b88c24f1427" path="/var/lib/kubelet/pods/327464b9-e1f2-4119-bc05-4b88c24f1427/volumes" Sep 10 15:13:00.230676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:00.230604 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8709a1c-1199-4240-95f2-209c738f707c" path="/var/lib/kubelet/pods/d8709a1c-1199-4240-95f2-209c738f707c/volumes" Sep 10 15:13:04.048446 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.048408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:04.374183 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.374089 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:04.441171 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.441137 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:04.651402 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651320 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk"] Sep 10 15:13:04.651743 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651730 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="327464b9-e1f2-4119-bc05-4b88c24f1427" containerName="payload-processing" Sep 10 15:13:04.651815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651745 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="327464b9-e1f2-4119-bc05-4b88c24f1427" containerName="payload-processing" Sep 10 15:13:04.651815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651776 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8709a1c-1199-4240-95f2-209c738f707c" containerName="payload-pre-processing" Sep 10 15:13:04.651815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651783 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8709a1c-1199-4240-95f2-209c738f707c" containerName="payload-pre-processing" Sep 10 15:13:04.651815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651798 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4c627c-d991-4b4c-92ea-8ed56b630c72" containerName="revoke-keys" Sep 10 15:13:04.651815 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651804 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4c627c-d991-4b4c-92ea-8ed56b630c72" containerName="revoke-keys" Sep 10 15:13:04.651970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651872 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8709a1c-1199-4240-95f2-209c738f707c" containerName="payload-pre-processing" Sep 10 15:13:04.651970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651881 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="327464b9-e1f2-4119-bc05-4b88c24f1427" containerName="payload-processing" Sep 10 15:13:04.651970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.651888 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4c627c-d991-4b4c-92ea-8ed56b630c72" containerName="revoke-keys" Sep 10 15:13:04.656839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.656819 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:04.659356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.659334 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:13:04.663527 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.663260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk"] Sep 10 15:13:04.710434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.710397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk\" (UID: \"ce2c8156-be99-43cd-8756-223330e6ed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:04.771902 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.771838 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:04.811853 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.811819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk\" (UID: \"ce2c8156-be99-43cd-8756-223330e6ed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:04.812497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.812478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk\" (UID: \"ce2c8156-be99-43cd-8756-223330e6ed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:04.970457 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:04.970367 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:05.261302 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:05.261279 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk"] Sep 10 15:13:05.449286 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:05.449248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" event={"ID":"ce2c8156-be99-43cd-8756-223330e6ed66","Type":"ContainerStarted","Data":"8be13d5128b2a986d8fc24e86ba557cceb530a7237c79f176c5ae6b331bca79e"} Sep 10 15:13:06.457484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:06.457443 2581 generic.go:356] "Generic (PLEG): container finished" podID="ce2c8156-be99-43cd-8756-223330e6ed66" containerID="ab696d4ccb4b312b677fb8bfdb7f808140213f8d50959bd16018d0319a2ae353" exitCode=0 Sep 10 15:13:06.458012 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:06.457538 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" event={"ID":"ce2c8156-be99-43cd-8756-223330e6ed66","Type":"ContainerDied","Data":"ab696d4ccb4b312b677fb8bfdb7f808140213f8d50959bd16018d0319a2ae353"} Sep 10 15:13:07.608660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:07.608636 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:07.738986 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:07.738960 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca\") pod \"ce2c8156-be99-43cd-8756-223330e6ed66\" (UID: \"ce2c8156-be99-43cd-8756-223330e6ed66\") " Sep 10 15:13:07.739348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:07.739326 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce2c8156-be99-43cd-8756-223330e6ed66" (UID: "ce2c8156-be99-43cd-8756-223330e6ed66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:07.839936 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:07.839907 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2c8156-be99-43cd-8756-223330e6ed66-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:08.471212 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:08.471184 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" Sep 10 15:13:08.471393 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:08.471177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk" event={"ID":"ce2c8156-be99-43cd-8756-223330e6ed66","Type":"ContainerDied","Data":"8be13d5128b2a986d8fc24e86ba557cceb530a7237c79f176c5ae6b331bca79e"} Sep 10 15:13:08.471393 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:08.471304 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8be13d5128b2a986d8fc24e86ba557cceb530a7237c79f176c5ae6b331bca79e" Sep 10 15:13:13.676483 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:13.676441 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:14.893581 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:14.893542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.470799 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.470732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.498354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.497665 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:13:15.498354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.497954 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" podUID="3abd227d-6f49-4951-af35-a5eae7bf925c" containerName="payload-processing" containerID="cri-o://d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a" gracePeriod=30 Sep 10 15:13:15.506399 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.506372 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:13:15.506661 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.506637 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" podUID="d7a02835-a6a2-4a5c-b058-74c146185ce1" containerName="payload-pre-processing" containerID="cri-o://8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad" gracePeriod=30 Sep 10 15:13:15.542053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.542019 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.614739 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.614701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.678169 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.678098 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk"] Sep 10 15:13:15.681253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.681219 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0a9de4-txfbk"] Sep 10 15:13:15.697909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.697850 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.762714 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.762670 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:15.799193 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.799169 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:13:15.803009 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.802984 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:13:15.914078 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914038 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume\") pod \"3abd227d-6f49-4951-af35-a5eae7bf925c\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " Sep 10 15:13:15.914472 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914089 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c92q\" (UniqueName: \"kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q\") pod \"3abd227d-6f49-4951-af35-a5eae7bf925c\" (UID: \"3abd227d-6f49-4951-af35-a5eae7bf925c\") " Sep 10 15:13:15.914472 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914130 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsrk9\" (UniqueName: \"kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9\") pod \"d7a02835-a6a2-4a5c-b058-74c146185ce1\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " Sep 10 15:13:15.914472 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914248 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume\") pod \"d7a02835-a6a2-4a5c-b058-74c146185ce1\" (UID: \"d7a02835-a6a2-4a5c-b058-74c146185ce1\") " Sep 10 15:13:15.914649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914496 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3abd227d-6f49-4951-af35-a5eae7bf925c" (UID: "3abd227d-6f49-4951-af35-a5eae7bf925c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:15.914649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914521 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7a02835-a6a2-4a5c-b058-74c146185ce1" (UID: "d7a02835-a6a2-4a5c-b058-74c146185ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:15.914649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914592 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a02835-a6a2-4a5c-b058-74c146185ce1-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:15.914649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.914604 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abd227d-6f49-4951-af35-a5eae7bf925c-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:15.916454 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.916433 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q" (OuterVolumeSpecName: "kube-api-access-6c92q") pod "3abd227d-6f49-4951-af35-a5eae7bf925c" (UID: "3abd227d-6f49-4951-af35-a5eae7bf925c"). InnerVolumeSpecName "kube-api-access-6c92q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:15.916553 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:15.916464 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9" (OuterVolumeSpecName: "kube-api-access-wsrk9") pod "d7a02835-a6a2-4a5c-b058-74c146185ce1" (UID: "d7a02835-a6a2-4a5c-b058-74c146185ce1"). InnerVolumeSpecName "kube-api-access-wsrk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:16.016001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.015904 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsrk9\" (UniqueName: \"kubernetes.io/projected/d7a02835-a6a2-4a5c-b058-74c146185ce1-kube-api-access-wsrk9\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:16.016001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.015942 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c92q\" (UniqueName: \"kubernetes.io/projected/3abd227d-6f49-4951-af35-a5eae7bf925c-kube-api-access-6c92q\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:16.029322 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.029289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:16.237961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.237928 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2c8156-be99-43cd-8756-223330e6ed66" path="/var/lib/kubelet/pods/ce2c8156-be99-43cd-8756-223330e6ed66/volumes" Sep 10 15:13:16.528266 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.528238 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7a02835-a6a2-4a5c-b058-74c146185ce1" containerID="8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad" exitCode=0 Sep 10 15:13:16.528434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.528295 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" Sep 10 15:13:16.528434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.528322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" event={"ID":"d7a02835-a6a2-4a5c-b058-74c146185ce1","Type":"ContainerDied","Data":"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad"} Sep 10 15:13:16.528434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.528379 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8" event={"ID":"d7a02835-a6a2-4a5c-b058-74c146185ce1","Type":"ContainerDied","Data":"06b9b07810d76c5d2c978588126b8c2a1eae8140c3634a2ce491caa7b071ab0e"} Sep 10 15:13:16.528434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.528410 2581 scope.go:117] "RemoveContainer" containerID="8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad" Sep 10 15:13:16.529745 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.529724 2581 generic.go:356] "Generic (PLEG): container finished" podID="3abd227d-6f49-4951-af35-a5eae7bf925c" containerID="d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a" exitCode=0 Sep 10 15:13:16.529836 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.529811 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" Sep 10 15:13:16.529885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.529850 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" event={"ID":"3abd227d-6f49-4951-af35-a5eae7bf925c","Type":"ContainerDied","Data":"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a"} Sep 10 15:13:16.529885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.529882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m" event={"ID":"3abd227d-6f49-4951-af35-a5eae7bf925c","Type":"ContainerDied","Data":"99db4b94f9f90b1f4bf738566352165ed189733027f9f297ac3fa389657c1012"} Sep 10 15:13:16.537889 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.537873 2581 scope.go:117] "RemoveContainer" containerID="8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad" Sep 10 15:13:16.538201 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:13:16.538182 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad\": container with ID starting with 8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad not found: ID does not exist" containerID="8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad" Sep 10 15:13:16.538260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.538213 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad"} err="failed to get container status \"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad\": rpc error: code = NotFound desc = could not find container \"8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad\": container with ID starting with 8f9dd408d39e3295220050f06b486f01d159463a0370ddcca93f2c1dff918aad not found: ID does not exist" Sep 10 15:13:16.538260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.538230 2581 scope.go:117] "RemoveContainer" containerID="d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a" Sep 10 15:13:16.547624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.547597 2581 scope.go:117] "RemoveContainer" containerID="d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a" Sep 10 15:13:16.547957 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:13:16.547934 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a\": container with ID starting with d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a not found: ID does not exist" containerID="d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a" Sep 10 15:13:16.548026 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.547966 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a"} err="failed to get container status \"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a\": rpc error: code = NotFound desc = could not find container \"d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a\": container with ID starting with d036b083fb38d4eedf3c9e2e7be434d46e5404c9b6c1930e4824295458be1b3a not found: ID does not exist" Sep 10 15:13:16.549157 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.549139 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:13:16.557356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.557328 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0a9de4-598f9bf6dc-npx7m"] Sep 10 15:13:16.570917 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.570889 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:13:16.574823 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.574803 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0a9de4-6d65c99d9d-bnsj8"] Sep 10 15:13:16.943679 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:16.943586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:18.229818 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:18.229782 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abd227d-6f49-4951-af35-a5eae7bf925c" path="/var/lib/kubelet/pods/3abd227d-6f49-4951-af35-a5eae7bf925c/volumes" Sep 10 15:13:18.230203 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:18.230135 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a02835-a6a2-4a5c-b058-74c146185ce1" path="/var/lib/kubelet/pods/d7a02835-a6a2-4a5c-b058-74c146185ce1/volumes" Sep 10 15:13:21.681249 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:21.681205 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:21.750431 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:21.750395 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:21.930577 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:21.930548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:22.302642 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.302603 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf"] Sep 10 15:13:22.303127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303110 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2c8156-be99-43cd-8756-223330e6ed66" containerName="revoke-keys" Sep 10 15:13:22.303127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303128 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2c8156-be99-43cd-8756-223330e6ed66" containerName="revoke-keys" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303140 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abd227d-6f49-4951-af35-a5eae7bf925c" containerName="payload-processing" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303149 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abd227d-6f49-4951-af35-a5eae7bf925c" containerName="payload-processing" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303161 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a02835-a6a2-4a5c-b058-74c146185ce1" containerName="payload-pre-processing" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303167 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a02835-a6a2-4a5c-b058-74c146185ce1" containerName="payload-pre-processing" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303235 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2c8156-be99-43cd-8756-223330e6ed66" containerName="revoke-keys" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303245 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abd227d-6f49-4951-af35-a5eae7bf925c" containerName="payload-processing" Sep 10 15:13:22.303287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.303252 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a02835-a6a2-4a5c-b058-74c146185ce1" containerName="payload-pre-processing" Sep 10 15:13:22.307727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.307700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:22.310251 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.310217 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:13:22.314471 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.314445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf"] Sep 10 15:13:22.408415 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.408375 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:22.475594 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.475552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf\" (UID: \"6c422141-bbff-47ef-913b-dd2a425179fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:22.576922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.576832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf\" (UID: \"6c422141-bbff-47ef-913b-dd2a425179fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:22.577505 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.577486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf\" (UID: \"6c422141-bbff-47ef-913b-dd2a425179fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:22.620808 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.620778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:22.898807 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:22.898772 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf"] Sep 10 15:13:22.900870 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:13:22.900840 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c422141_bbff_47ef_913b_dd2a425179fa.slice/crio-457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50 WatchSource:0}: Error finding container 457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50: Status 404 returned error can't find the container with id 457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50 Sep 10 15:13:23.572114 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:23.572033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" event={"ID":"6c422141-bbff-47ef-913b-dd2a425179fa","Type":"ContainerStarted","Data":"2c106d6e5b819e00b40ae76fa977b58c232a4ae63393768989fc23fc4244ecd2"} Sep 10 15:13:23.572114 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:23.572085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" event={"ID":"6c422141-bbff-47ef-913b-dd2a425179fa","Type":"ContainerStarted","Data":"457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50"} Sep 10 15:13:23.587528 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:23.587474 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" podStartSLOduration=1.5874589650000002 podStartE2EDuration="1.587458965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:13:23.586112708 +0000 UTC m=+1557.880414839" watchObservedRunningTime="2026-09-10 15:13:23.587458965 +0000 UTC m=+1557.881761093" Sep 10 15:13:24.579971 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:24.579932 2581 generic.go:356] "Generic (PLEG): container finished" podID="6c422141-bbff-47ef-913b-dd2a425179fa" containerID="2c106d6e5b819e00b40ae76fa977b58c232a4ae63393768989fc23fc4244ecd2" exitCode=0 Sep 10 15:13:24.580354 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:24.580021 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" event={"ID":"6c422141-bbff-47ef-913b-dd2a425179fa","Type":"ContainerDied","Data":"2c106d6e5b819e00b40ae76fa977b58c232a4ae63393768989fc23fc4244ecd2"} Sep 10 15:13:25.730364 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:25.730336 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:25.911052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:25.910960 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca\") pod \"6c422141-bbff-47ef-913b-dd2a425179fa\" (UID: \"6c422141-bbff-47ef-913b-dd2a425179fa\") " Sep 10 15:13:25.911374 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:25.911345 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c422141-bbff-47ef-913b-dd2a425179fa" (UID: "6c422141-bbff-47ef-913b-dd2a425179fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:26.012019 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:26.011980 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c422141-bbff-47ef-913b-dd2a425179fa-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:26.593919 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:26.593893 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" Sep 10 15:13:26.594056 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:26.593925 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf" event={"ID":"6c422141-bbff-47ef-913b-dd2a425179fa","Type":"ContainerDied","Data":"457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50"} Sep 10 15:13:26.594056 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:26.594021 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457fb2c8d40fd88aa23c344fec95e2d34f969d0344b0d30b27bf43ccdaa22b50" Sep 10 15:13:32.822111 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.822072 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:32.845939 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.844814 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:13:32.849192 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.849081 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" podUID="5cda971e-0791-4d52-a15b-d95e2ef2648d" containerName="payload-processing" containerID="cri-o://f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210" gracePeriod=30 Sep 10 15:13:32.853913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.853876 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:13:32.854862 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.854188 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" podUID="6b5e50af-c2af-49bd-816b-3eeadec07c10" containerName="payload-pre-processing" containerID="cri-o://fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737" gracePeriod=30 Sep 10 15:13:32.893214 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.893029 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:32.982367 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:32.982338 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:33.006226 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.005263 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf"] Sep 10 15:13:33.008029 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.007997 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1f985c-7cdlf"] Sep 10 15:13:33.057513 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.057484 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:33.149808 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.149781 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:13:33.152876 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.152852 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:13:33.167083 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.166974 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume\") pod \"6b5e50af-c2af-49bd-816b-3eeadec07c10\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " Sep 10 15:13:33.167083 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167023 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxfc4\" (UniqueName: \"kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4\") pod \"6b5e50af-c2af-49bd-816b-3eeadec07c10\" (UID: \"6b5e50af-c2af-49bd-816b-3eeadec07c10\") " Sep 10 15:13:33.167083 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167058 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume\") pod \"5cda971e-0791-4d52-a15b-d95e2ef2648d\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " Sep 10 15:13:33.167362 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167090 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz6vq\" (UniqueName: \"kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq\") pod \"5cda971e-0791-4d52-a15b-d95e2ef2648d\" (UID: \"5cda971e-0791-4d52-a15b-d95e2ef2648d\") " Sep 10 15:13:33.167524 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167430 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b5e50af-c2af-49bd-816b-3eeadec07c10" (UID: "6b5e50af-c2af-49bd-816b-3eeadec07c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:33.167616 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167584 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5e50af-c2af-49bd-816b-3eeadec07c10-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:33.167920 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.167897 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cda971e-0791-4d52-a15b-d95e2ef2648d" (UID: "5cda971e-0791-4d52-a15b-d95e2ef2648d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:33.170243 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.170216 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4" (OuterVolumeSpecName: "kube-api-access-zxfc4") pod "6b5e50af-c2af-49bd-816b-3eeadec07c10" (UID: "6b5e50af-c2af-49bd-816b-3eeadec07c10"). InnerVolumeSpecName "kube-api-access-zxfc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:33.170360 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.170230 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq" (OuterVolumeSpecName: "kube-api-access-pz6vq") pod "5cda971e-0791-4d52-a15b-d95e2ef2648d" (UID: "5cda971e-0791-4d52-a15b-d95e2ef2648d"). InnerVolumeSpecName "kube-api-access-pz6vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:33.268502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.268462 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfc4\" (UniqueName: \"kubernetes.io/projected/6b5e50af-c2af-49bd-816b-3eeadec07c10-kube-api-access-zxfc4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:33.268502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.268490 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cda971e-0791-4d52-a15b-d95e2ef2648d-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:33.268502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.268502 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz6vq\" (UniqueName: \"kubernetes.io/projected/5cda971e-0791-4d52-a15b-d95e2ef2648d-kube-api-access-pz6vq\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:13:33.365702 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.365663 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:33.635728 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.635687 2581 generic.go:356] "Generic (PLEG): container finished" podID="6b5e50af-c2af-49bd-816b-3eeadec07c10" containerID="fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737" exitCode=0 Sep 10 15:13:33.635973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.635743 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" Sep 10 15:13:33.635973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.635803 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" event={"ID":"6b5e50af-c2af-49bd-816b-3eeadec07c10","Type":"ContainerDied","Data":"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737"} Sep 10 15:13:33.635973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.635860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j" event={"ID":"6b5e50af-c2af-49bd-816b-3eeadec07c10","Type":"ContainerDied","Data":"ed301aa73f0d08d614ea2f95e13e161b8f62bc2d5cd2d57faa9fcd9f71dc40ce"} Sep 10 15:13:33.635973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.635890 2581 scope.go:117] "RemoveContainer" containerID="fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737" Sep 10 15:13:33.637277 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.637247 2581 generic.go:356] "Generic (PLEG): container finished" podID="5cda971e-0791-4d52-a15b-d95e2ef2648d" containerID="f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210" exitCode=0 Sep 10 15:13:33.637414 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.637288 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" Sep 10 15:13:33.637414 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.637297 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" event={"ID":"5cda971e-0791-4d52-a15b-d95e2ef2648d","Type":"ContainerDied","Data":"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210"} Sep 10 15:13:33.637414 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.637332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z" event={"ID":"5cda971e-0791-4d52-a15b-d95e2ef2648d","Type":"ContainerDied","Data":"f97554a5edfba8a4768d651607e43db0e2a80a43798da226d03974ce1e754358"} Sep 10 15:13:33.646159 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.646141 2581 scope.go:117] "RemoveContainer" containerID="fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737" Sep 10 15:13:33.646579 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:13:33.646446 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737\": container with ID starting with fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737 not found: ID does not exist" containerID="fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737" Sep 10 15:13:33.646579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.646483 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737"} err="failed to get container status \"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737\": rpc error: code = NotFound desc = could not find container \"fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737\": container with ID starting with fb140364517f559e6331eefe7d8d84437714d3c2a5831a3342dae259007ed737 not found: ID does not exist" Sep 10 15:13:33.646579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.646499 2581 scope.go:117] "RemoveContainer" containerID="f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210" Sep 10 15:13:33.655190 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.655173 2581 scope.go:117] "RemoveContainer" containerID="f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210" Sep 10 15:13:33.655521 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:13:33.655500 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210\": container with ID starting with f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210 not found: ID does not exist" containerID="f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210" Sep 10 15:13:33.655599 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.655531 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210"} err="failed to get container status \"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210\": rpc error: code = NotFound desc = could not find container \"f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210\": container with ID starting with f4b6bcd31cfd5a4023cfa10e95dc5b8acf20f0f74bb191731adc0b86df8d6210 not found: ID does not exist" Sep 10 15:13:33.658579 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.658554 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:13:33.662024 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.662003 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1f985c-6dc6cfc854-2fp6j"] Sep 10 15:13:33.672171 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.672147 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:13:33.676236 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:33.676205 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1f985c-6465bf9f59-dmh7z"] Sep 10 15:13:34.230253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:34.230222 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cda971e-0791-4d52-a15b-d95e2ef2648d" path="/var/lib/kubelet/pods/5cda971e-0791-4d52-a15b-d95e2ef2648d/volumes" Sep 10 15:13:34.230610 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:34.230556 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5e50af-c2af-49bd-816b-3eeadec07c10" path="/var/lib/kubelet/pods/6b5e50af-c2af-49bd-816b-3eeadec07c10/volumes" Sep 10 15:13:34.230861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:34.230847 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c422141-bbff-47ef-913b-dd2a425179fa" path="/var/lib/kubelet/pods/6c422141-bbff-47ef-913b-dd2a425179fa/volumes" Sep 10 15:13:39.454822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:39.454788 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:39.554879 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:39.554836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:39.722179 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:39.722093 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:57.473319 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:57.473281 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:58.600722 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:58.600683 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:58.996913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:58.996881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:59.298181 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:59.298069 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:13:59.516933 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:13:59.516903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:02.704696 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:02.704658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:02.801149 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:02.801101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:03.972746 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:03.972712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:04.073016 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.072983 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:04.550548 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.550508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:04.652544 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.652503 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:04.661142 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:14:04.661516 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661504 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5e50af-c2af-49bd-816b-3eeadec07c10" containerName="payload-pre-processing" Sep 10 15:14:04.661559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661518 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5e50af-c2af-49bd-816b-3eeadec07c10" containerName="payload-pre-processing" Sep 10 15:14:04.661559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661535 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c422141-bbff-47ef-913b-dd2a425179fa" containerName="revoke-keys" Sep 10 15:14:04.661559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661540 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c422141-bbff-47ef-913b-dd2a425179fa" containerName="revoke-keys" Sep 10 15:14:04.661559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661547 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cda971e-0791-4d52-a15b-d95e2ef2648d" containerName="payload-processing" Sep 10 15:14:04.661559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661552 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cda971e-0791-4d52-a15b-d95e2ef2648d" containerName="payload-processing" Sep 10 15:14:04.661769 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661608 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c422141-bbff-47ef-913b-dd2a425179fa" containerName="revoke-keys" Sep 10 15:14:04.661769 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661615 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5e50af-c2af-49bd-816b-3eeadec07c10" containerName="payload-pre-processing" Sep 10 15:14:04.661769 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.661621 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cda971e-0791-4d52-a15b-d95e2ef2648d" containerName="payload-processing" Sep 10 15:14:04.664683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.664663 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.667161 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.667145 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-13ee06-dockercfg-c8p8v\"" Sep 10 15:14:04.667698 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.667677 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-13ee06\"" Sep 10 15:14:04.676002 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.675979 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:14:04.688152 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.688123 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:14:04.691858 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.691834 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.707368 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.707338 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:14:04.753257 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.753222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.753257 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.753259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ds6l\" (UniqueName: \"kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.854596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.854515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.854596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.854553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ds6l\" (UniqueName: \"kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.854814 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.854598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.854814 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.854650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm9p\" (UniqueName: \"kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.855292 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.855272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.863110 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.863084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ds6l\" (UniqueName: \"kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l\") pod \"payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:04.955580 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.955543 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.955823 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.955638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgm9p\" (UniqueName: \"kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.956287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.956268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.964405 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.964383 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgm9p\" (UniqueName: \"kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p\") pod \"payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:04.976227 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:04.976205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:05.006162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.006127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:05.256190 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.256164 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:14:05.257737 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:14:05.257691 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2df2bcc_a874_4a8e_9b0f_0d329df610de.slice/crio-2358426c3abe379509ee4f57cbb95f694b839b26438cc8ec1ebcf34be17966b8 WatchSource:0}: Error finding container 2358426c3abe379509ee4f57cbb95f694b839b26438cc8ec1ebcf34be17966b8: Status 404 returned error can't find the container with id 2358426c3abe379509ee4f57cbb95f694b839b26438cc8ec1ebcf34be17966b8 Sep 10 15:14:05.301128 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.301102 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:14:05.303584 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:14:05.303560 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e6efd4_4c86_4065_8dba_3db82d0da337.slice/crio-3fa739db1acf76975a914fffff4ca7dfb0e4c8027d0164e8cf3b9da14987c4e0 WatchSource:0}: Error finding container 3fa739db1acf76975a914fffff4ca7dfb0e4c8027d0164e8cf3b9da14987c4e0: Status 404 returned error can't find the container with id 3fa739db1acf76975a914fffff4ca7dfb0e4c8027d0164e8cf3b9da14987c4e0 Sep 10 15:14:05.819953 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.819911 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" event={"ID":"e6e6efd4-4c86-4065-8dba-3db82d0da337","Type":"ContainerStarted","Data":"753b47d16b8c98bb37c4087904955c60b8ed768b35228ffe61c76b51ef4052e7"} Sep 10 15:14:05.820162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.819976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" event={"ID":"e6e6efd4-4c86-4065-8dba-3db82d0da337","Type":"ContainerStarted","Data":"3fa739db1acf76975a914fffff4ca7dfb0e4c8027d0164e8cf3b9da14987c4e0"} Sep 10 15:14:05.820356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.820332 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:05.821818 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.821791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" event={"ID":"c2df2bcc-a874-4a8e-9b0f-0d329df610de","Type":"ContainerStarted","Data":"af165bc78c9ec3f2ffd2d25f9f1cf4d03ef955046d521a8591ca381deb6146f2"} Sep 10 15:14:05.821967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.821838 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" event={"ID":"c2df2bcc-a874-4a8e-9b0f-0d329df610de","Type":"ContainerStarted","Data":"2358426c3abe379509ee4f57cbb95f694b839b26438cc8ec1ebcf34be17966b8"} Sep 10 15:14:05.821967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.821894 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:05.841910 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.841811 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" podStartSLOduration=1.841790904 podStartE2EDuration="1.841790904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:05.839119443 +0000 UTC m=+1600.133421574" watchObservedRunningTime="2026-09-10 15:14:05.841790904 +0000 UTC m=+1600.136093036" Sep 10 15:14:05.858723 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:05.858670 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" podStartSLOduration=1.858654451 podStartE2EDuration="1.858654451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:05.857675619 +0000 UTC m=+1600.151977749" watchObservedRunningTime="2026-09-10 15:14:05.858654451 +0000 UTC m=+1600.152956579" Sep 10 15:14:16.830775 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:16.830663 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:14:16.830775 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:16.830748 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:14:21.906922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:21.906879 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:24.910703 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:24.910658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:25.010928 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:25.010894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:27.661475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:27.661441 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:28.350361 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:28.350330 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:28.671669 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:28.671571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:29.039530 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:29.039497 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:31.884311 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:31.884263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:31.975812 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:31.975772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.197888 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.197854 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.253796 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.253736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.370810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.370779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.655453 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.655417 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.758099 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.758063 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:33.765442 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.765414 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:14:33.770053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.770036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.772287 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.772268 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6333de\"" Sep 10 15:14:33.772371 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.772314 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6333de-dockercfg-kc2kl\"" Sep 10 15:14:33.780787 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.780742 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:14:33.811008 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.810972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.811193 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.811062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm54r\" (UniqueName: \"kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.814159 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.814134 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:14:33.818009 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.817991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:33.827585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.827563 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:14:33.911949 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.911867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.911949 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.911909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmx2z\" (UniqueName: \"kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:33.912176 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.912043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm54r\" (UniqueName: \"kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.912176 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.912104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:33.912605 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.912587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:33.920670 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:33.920642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm54r\" (UniqueName: \"kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r\") pod \"payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:34.013059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.013027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmx2z\" (UniqueName: \"kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.013230 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.013122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.013839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.013819 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.022105 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.022072 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmx2z\" (UniqueName: \"kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z\") pod \"payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.083937 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.083899 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:34.130567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.130514 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.357547 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.357501 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:14:34.422864 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.422829 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:14:34.424587 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:14:34.424560 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624b0bd0_2a1b_44fd_803f_563438287437.slice/crio-419cdc51e59b675aab38d42d09ef36c9f2ed7907e37226fe21d11b4e3986fb1b WatchSource:0}: Error finding container 419cdc51e59b675aab38d42d09ef36c9f2ed7907e37226fe21d11b4e3986fb1b: Status 404 returned error can't find the container with id 419cdc51e59b675aab38d42d09ef36c9f2ed7907e37226fe21d11b4e3986fb1b Sep 10 15:14:34.986048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.986013 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" event={"ID":"624b0bd0-2a1b-44fd-803f-563438287437","Type":"ContainerStarted","Data":"d53a6ea045e818644caa4204b24dc347d8bbda046e592cedc251fd73fe5c2824"} Sep 10 15:14:34.986356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.986167 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:34.986356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.986214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" event={"ID":"624b0bd0-2a1b-44fd-803f-563438287437","Type":"ContainerStarted","Data":"419cdc51e59b675aab38d42d09ef36c9f2ed7907e37226fe21d11b4e3986fb1b"} Sep 10 15:14:34.987677 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.987656 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" event={"ID":"795b45ba-6a7d-4481-a08e-ba50f5c76514","Type":"ContainerStarted","Data":"53883411940c1994e9e4ef407165d9324bd2c63b8b99110ad701ae3863f36a12"} Sep 10 15:14:34.987828 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.987696 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" event={"ID":"795b45ba-6a7d-4481-a08e-ba50f5c76514","Type":"ContainerStarted","Data":"bd819b7fd7cdb1c3e3ae41485b00c72166a9d60b67ef0f8ae976468e6dab1eb0"} Sep 10 15:14:34.987828 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:34.987730 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:35.002312 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:35.002245 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" podStartSLOduration=2.002224673 podStartE2EDuration="2.002224673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:35.001913782 +0000 UTC m=+1629.296215913" watchObservedRunningTime="2026-09-10 15:14:35.002224673 +0000 UTC m=+1629.296526798" Sep 10 15:14:35.018030 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:35.017982 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" podStartSLOduration=2.017964385 podStartE2EDuration="2.017964385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:35.01702663 +0000 UTC m=+1629.311328751" watchObservedRunningTime="2026-09-10 15:14:35.017964385 +0000 UTC m=+1629.312266515" Sep 10 15:14:45.761235 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:45.761197 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:45.820663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:45.820626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:45.996318 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:45.996292 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:14:45.996512 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:45.996366 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:14:56.844888 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:56.844824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:14:59.683834 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:14:59.683793 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:00.142160 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.142119 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb"] Sep 10 15:15:00.145726 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.145699 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms"] Sep 10 15:15:00.145922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.145834 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" Sep 10 15:15:00.148939 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.148921 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd"] Sep 10 15:15:00.149072 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.149034 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" Sep 10 15:15:00.152379 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.152260 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" Sep 10 15:15:00.156980 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.156945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms"] Sep 10 15:15:00.160403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.160369 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb"] Sep 10 15:15:00.163344 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.163317 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd"] Sep 10 15:15:00.165147 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.165124 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:15:00.170428 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.170396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" Sep 10 15:15:00.171140 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.171115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" Sep 10 15:15:00.180697 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.180669 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" Sep 10 15:15:00.457340 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.457306 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms"] Sep 10 15:15:00.459944 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:15:00.459772 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78ecda3_7e57_4265_92ff_1f60ac824f0b.slice/crio-cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1 WatchSource:0}: Error finding container cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1: Status 404 returned error can't find the container with id cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1 Sep 10 15:15:00.460954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.460926 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb"] Sep 10 15:15:00.462570 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:15:00.462546 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04f12ee_0041_4239_86a4_1371639e0573.slice/crio-bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92 WatchSource:0}: Error finding container bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92: Status 404 returned error can't find the container with id bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92 Sep 10 15:15:00.479293 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:00.479225 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd"] Sep 10 15:15:00.483848 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:15:00.483821 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91304ed2_1a53_4f96_af5a_81bc51d52ce9.slice/crio-695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a WatchSource:0}: Error finding container 695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a: Status 404 returned error can't find the container with id 695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a Sep 10 15:15:01.138305 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.138264 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" event={"ID":"a04f12ee-0041-4239-86a4-1371639e0573","Type":"ContainerStarted","Data":"731cda20501bdd4911ec87b3ced38525a5989974ff55f1a9aa1e9d207f601552"} Sep 10 15:15:01.138775 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.138363 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" event={"ID":"a04f12ee-0041-4239-86a4-1371639e0573","Type":"ContainerStarted","Data":"bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92"} Sep 10 15:15:01.139822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.139799 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" event={"ID":"a78ecda3-7e57-4265-92ff-1f60ac824f0b","Type":"ContainerStarted","Data":"c308bbb510f1f180dbafe5ae3332dc4b3ba0687dbabf8f2e7043607a899d21e8"} Sep 10 15:15:01.139949 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.139839 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" event={"ID":"a78ecda3-7e57-4265-92ff-1f60ac824f0b","Type":"ContainerStarted","Data":"cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1"} Sep 10 15:15:01.141115 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.141093 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" event={"ID":"91304ed2-1a53-4f96-af5a-81bc51d52ce9","Type":"ContainerStarted","Data":"2e5bbeb05866ea252ec0e25ffe348388b034e9232b39feb15223120723231bff"} Sep 10 15:15:01.141220 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.141131 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" event={"ID":"91304ed2-1a53-4f96-af5a-81bc51d52ce9","Type":"ContainerStarted","Data":"695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a"} Sep 10 15:15:01.153334 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.153291 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" podStartSLOduration=1.153275177 podStartE2EDuration="1.153275177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:01.151257976 +0000 UTC m=+1655.445560105" watchObservedRunningTime="2026-09-10 15:15:01.153275177 +0000 UTC m=+1655.447577306" Sep 10 15:15:01.166328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.166268 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" podStartSLOduration=1.166250575 podStartE2EDuration="1.166250575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:01.166071039 +0000 UTC m=+1655.460373180" watchObservedRunningTime="2026-09-10 15:15:01.166250575 +0000 UTC m=+1655.460552705" Sep 10 15:15:01.180274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.180220 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" podStartSLOduration=1.1802030669999999 podStartE2EDuration="1.180203067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:01.178512614 +0000 UTC m=+1655.472814743" watchObservedRunningTime="2026-09-10 15:15:01.180203067 +0000 UTC m=+1655.474505196" Sep 10 15:15:01.835896 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.835812 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:15:01.958863 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:01.958826 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2_624b0bd0-2a1b-44fd-803f-563438287437/payload-processing/0.log" Sep 10 15:15:02.150473 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.150387 2581 generic.go:356] "Generic (PLEG): container finished" podID="a78ecda3-7e57-4265-92ff-1f60ac824f0b" containerID="c308bbb510f1f180dbafe5ae3332dc4b3ba0687dbabf8f2e7043607a899d21e8" exitCode=0 Sep 10 15:15:02.150909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.150476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" event={"ID":"a78ecda3-7e57-4265-92ff-1f60ac824f0b","Type":"ContainerDied","Data":"c308bbb510f1f180dbafe5ae3332dc4b3ba0687dbabf8f2e7043607a899d21e8"} Sep 10 15:15:02.152051 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.152027 2581 generic.go:356] "Generic (PLEG): container finished" podID="91304ed2-1a53-4f96-af5a-81bc51d52ce9" containerID="2e5bbeb05866ea252ec0e25ffe348388b034e9232b39feb15223120723231bff" exitCode=0 Sep 10 15:15:02.152138 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.152105 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" event={"ID":"91304ed2-1a53-4f96-af5a-81bc51d52ce9","Type":"ContainerDied","Data":"2e5bbeb05866ea252ec0e25ffe348388b034e9232b39feb15223120723231bff"} Sep 10 15:15:02.153641 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.153618 2581 generic.go:356] "Generic (PLEG): container finished" podID="a04f12ee-0041-4239-86a4-1371639e0573" containerID="731cda20501bdd4911ec87b3ced38525a5989974ff55f1a9aa1e9d207f601552" exitCode=0 Sep 10 15:15:02.153731 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:02.153678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" event={"ID":"a04f12ee-0041-4239-86a4-1371639e0573","Type":"ContainerDied","Data":"731cda20501bdd4911ec87b3ced38525a5989974ff55f1a9aa1e9d207f601552"} Sep 10 15:15:03.298403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:03.298374 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" Sep 10 15:15:03.340253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:03.340226 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" Sep 10 15:15:03.382536 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:03.382509 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" Sep 10 15:15:04.166195 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.166165 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" Sep 10 15:15:04.166195 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.166188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd" event={"ID":"91304ed2-1a53-4f96-af5a-81bc51d52ce9","Type":"ContainerDied","Data":"695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a"} Sep 10 15:15:04.166463 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.166235 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695fc5a91b146aaef7c1c387f856794aa6d6a7f943248f75c845849790c4176a" Sep 10 15:15:04.167563 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.167544 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" Sep 10 15:15:04.167634 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.167583 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb" event={"ID":"a04f12ee-0041-4239-86a4-1371639e0573","Type":"ContainerDied","Data":"bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92"} Sep 10 15:15:04.167695 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.167631 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc3ebc36e08dd6e66f736e35805ed3bbdc4f298836314eb3494b22aedbc6a92" Sep 10 15:15:04.168866 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.168845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" event={"ID":"a78ecda3-7e57-4265-92ff-1f60ac824f0b","Type":"ContainerDied","Data":"cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1"} Sep 10 15:15:04.168962 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.168878 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba703d7efe5f366a6904a08045935db56d5f894a09dcb48a853213790afcec1" Sep 10 15:15:04.168962 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.168877 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms" Sep 10 15:15:04.352970 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:04.352939 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:13.179093 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:13.179056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:26.630032 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:26.629995 2581 scope.go:117] "RemoveContainer" containerID="9e30b8ef2448309c04581e93b3d8b9cb2160cd695f47b02ff325f63e3da7d28b" Sep 10 15:15:42.644169 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:42.644132 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r_e6e6efd4-4c86-4065-8dba-3db82d0da337/payload-processing/0.log" Sep 10 15:15:42.773061 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:42.773031 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:15:42.899333 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:42.899248 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2_624b0bd0-2a1b-44fd-803f-563438287437/payload-processing/0.log" Sep 10 15:15:43.157100 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:43.157009 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:43.451683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:43.451586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:43.839459 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:43.839425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:46.291843 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:46.291812 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:46.380282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:46.380251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:47.466952 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:47.466897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:47.567372 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:47.567337 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:47.667257 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:47.667224 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:47.925420 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:47.925387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:48.024415 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.024369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:15:48.050792 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.050737 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:15:48.051265 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051246 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04f12ee-0041-4239-86a4-1371639e0573" containerName="cleanup" Sep 10 15:15:48.051356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051268 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04f12ee-0041-4239-86a4-1371639e0573" containerName="cleanup" Sep 10 15:15:48.051356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051294 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78ecda3-7e57-4265-92ff-1f60ac824f0b" containerName="cleanup" Sep 10 15:15:48.051356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051302 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78ecda3-7e57-4265-92ff-1f60ac824f0b" containerName="cleanup" Sep 10 15:15:48.051356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051314 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91304ed2-1a53-4f96-af5a-81bc51d52ce9" containerName="cleanup" Sep 10 15:15:48.051356 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051323 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91304ed2-1a53-4f96-af5a-81bc51d52ce9" containerName="cleanup" Sep 10 15:15:48.051603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051448 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91304ed2-1a53-4f96-af5a-81bc51d52ce9" containerName="cleanup" Sep 10 15:15:48.051603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051463 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04f12ee-0041-4239-86a4-1371639e0573" containerName="cleanup" Sep 10 15:15:48.051603 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.051474 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78ecda3-7e57-4265-92ff-1f60ac824f0b" containerName="cleanup" Sep 10 15:15:48.055935 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.055913 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.058478 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.058458 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-06cc0e-dockercfg-4hpdb\"" Sep 10 15:15:48.058478 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.058472 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-06cc0e\"" Sep 10 15:15:48.064793 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.064737 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:15:48.076048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.076023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.076190 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.076086 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrjn\" (UniqueName: \"kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.124061 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.124031 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:15:48.127489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.127473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.136735 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.136709 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:15:48.177280 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.177190 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.177280 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.177262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768pc\" (UniqueName: \"kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.177484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.177368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrjn\" (UniqueName: \"kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.177484 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.177408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.178025 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.178009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.186234 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.186201 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrjn\" (UniqueName: \"kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn\") pod \"payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.278001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.277968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.278177 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.278061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768pc\" (UniqueName: \"kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.278683 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.278663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.286885 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.286853 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-768pc\" (UniqueName: \"kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc\") pod \"payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.367489 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.367451 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:48.439104 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.439018 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:48.646730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.646706 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:15:48.648973 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:15:48.648947 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe74a39c_bfdc_44d2_8a14_ce8094bba24a.slice/crio-37e8c4ae9d83003abe8715b4880b9c059f0e33859c326dce8680d6ab0653f9db WatchSource:0}: Error finding container 37e8c4ae9d83003abe8715b4880b9c059f0e33859c326dce8680d6ab0653f9db: Status 404 returned error can't find the container with id 37e8c4ae9d83003abe8715b4880b9c059f0e33859c326dce8680d6ab0653f9db Sep 10 15:15:48.717432 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:48.717404 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:15:48.719639 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:15:48.719612 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda897a7f3_f982_47b0_a531_bc87a2b09dad.slice/crio-1f644ace1075e5ed9928c7cef2eab27ccde8aaf8e074d780ec2cb7377f784e5c WatchSource:0}: Error finding container 1f644ace1075e5ed9928c7cef2eab27ccde8aaf8e074d780ec2cb7377f784e5c: Status 404 returned error can't find the container with id 1f644ace1075e5ed9928c7cef2eab27ccde8aaf8e074d780ec2cb7377f784e5c Sep 10 15:15:49.439745 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.439699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" event={"ID":"a897a7f3-f982-47b0-a531-bc87a2b09dad","Type":"ContainerStarted","Data":"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0"} Sep 10 15:15:49.439964 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.439803 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:15:49.439964 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.439828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" event={"ID":"a897a7f3-f982-47b0-a531-bc87a2b09dad","Type":"ContainerStarted","Data":"1f644ace1075e5ed9928c7cef2eab27ccde8aaf8e074d780ec2cb7377f784e5c"} Sep 10 15:15:49.441470 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.441444 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" event={"ID":"be74a39c-bfdc-44d2-8a14-ce8094bba24a","Type":"ContainerStarted","Data":"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d"} Sep 10 15:15:49.441583 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.441493 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" event={"ID":"be74a39c-bfdc-44d2-8a14-ce8094bba24a","Type":"ContainerStarted","Data":"37e8c4ae9d83003abe8715b4880b9c059f0e33859c326dce8680d6ab0653f9db"} Sep 10 15:15:49.441583 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.441523 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:15:49.456230 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.456173 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" podStartSLOduration=1.456155491 podStartE2EDuration="1.456155491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:49.455096757 +0000 UTC m=+1703.749398887" watchObservedRunningTime="2026-09-10 15:15:49.456155491 +0000 UTC m=+1703.750457621" Sep 10 15:15:49.472851 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:15:49.472789 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" podStartSLOduration=1.472749747 podStartE2EDuration="1.472749747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:49.47027275 +0000 UTC m=+1703.764574903" watchObservedRunningTime="2026-09-10 15:15:49.472749747 +0000 UTC m=+1703.767051875" Sep 10 15:16:00.449245 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:00.449211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:16:00.449637 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:00.449377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:16:11.216154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:11.216113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:11.890367 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:11.890321 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5"] Sep 10 15:16:11.894044 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:11.894019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:11.896494 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:11.896470 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:16:11.901747 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:11.901714 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5"] Sep 10 15:16:12.006535 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.006498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5\" (UID: \"c03f8df6-132e-4d14-b9e9-009a03e33941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:12.107396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.107355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5\" (UID: \"c03f8df6-132e-4d14-b9e9-009a03e33941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:12.108045 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.108023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5\" (UID: \"c03f8df6-132e-4d14-b9e9-009a03e33941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:12.207645 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.207556 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:12.482496 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.482470 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5"] Sep 10 15:16:12.484415 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:16:12.484387 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f8df6_132e_4d14_b9e9_009a03e33941.slice/crio-d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d WatchSource:0}: Error finding container d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d: Status 404 returned error can't find the container with id d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d Sep 10 15:16:12.580875 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:12.580837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" event={"ID":"c03f8df6-132e-4d14-b9e9-009a03e33941","Type":"ContainerStarted","Data":"d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d"} Sep 10 15:16:13.588357 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:13.588268 2581 generic.go:356] "Generic (PLEG): container finished" podID="c03f8df6-132e-4d14-b9e9-009a03e33941" containerID="8d0347041b9e33a61aed2f62eb45f3f891ce8c6fbe8f410e751d32b5b94fcb5f" exitCode=0 Sep 10 15:16:13.588357 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:13.588347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" event={"ID":"c03f8df6-132e-4d14-b9e9-009a03e33941","Type":"ContainerDied","Data":"8d0347041b9e33a61aed2f62eb45f3f891ce8c6fbe8f410e751d32b5b94fcb5f"} Sep 10 15:16:14.738204 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:14.738181 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:14.831962 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:14.831926 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca\") pod \"c03f8df6-132e-4d14-b9e9-009a03e33941\" (UID: \"c03f8df6-132e-4d14-b9e9-009a03e33941\") " Sep 10 15:16:14.832289 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:14.832266 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c03f8df6-132e-4d14-b9e9-009a03e33941" (UID: "c03f8df6-132e-4d14-b9e9-009a03e33941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:14.933503 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:14.933415 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f8df6-132e-4d14-b9e9-009a03e33941-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:15.606944 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:15.606910 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" Sep 10 15:16:15.606944 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:15.606932 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5" event={"ID":"c03f8df6-132e-4d14-b9e9-009a03e33941","Type":"ContainerDied","Data":"d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d"} Sep 10 15:16:15.607173 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:15.607017 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d430510e3d211ba80a61a49789d9416de279e28773dd1a24782115143cdfa45d" Sep 10 15:16:18.027093 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.027060 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:18.050026 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.049992 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:16:18.050267 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.050242 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" podUID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" containerName="payload-processing" containerID="cri-o://d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d" gracePeriod=30 Sep 10 15:16:18.059166 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.059135 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:16:18.059504 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.059476 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" podUID="a897a7f3-f982-47b0-a531-bc87a2b09dad" containerName="payload-pre-processing" containerID="cri-o://616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0" gracePeriod=30 Sep 10 15:16:18.093031 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.092908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:18.164710 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.164665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:18.232856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.232746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:18.233784 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.233739 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5"] Sep 10 15:16:18.233899 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.233794 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-06cc0e-5rjr5"] Sep 10 15:16:18.361731 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.361704 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:16:18.365039 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.365011 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:16:18.462667 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.462635 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrrjn\" (UniqueName: \"kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn\") pod \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " Sep 10 15:16:18.462909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.462707 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume\") pod \"a897a7f3-f982-47b0-a531-bc87a2b09dad\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " Sep 10 15:16:18.462909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.462749 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume\") pod \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\" (UID: \"be74a39c-bfdc-44d2-8a14-ce8094bba24a\") " Sep 10 15:16:18.462909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.462804 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-768pc\" (UniqueName: \"kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc\") pod \"a897a7f3-f982-47b0-a531-bc87a2b09dad\" (UID: \"a897a7f3-f982-47b0-a531-bc87a2b09dad\") " Sep 10 15:16:18.463132 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.463102 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a897a7f3-f982-47b0-a531-bc87a2b09dad" (UID: "a897a7f3-f982-47b0-a531-bc87a2b09dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:18.463190 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.463129 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be74a39c-bfdc-44d2-8a14-ce8094bba24a" (UID: "be74a39c-bfdc-44d2-8a14-ce8094bba24a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:18.465070 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.465050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc" (OuterVolumeSpecName: "kube-api-access-768pc") pod "a897a7f3-f982-47b0-a531-bc87a2b09dad" (UID: "a897a7f3-f982-47b0-a531-bc87a2b09dad"). InnerVolumeSpecName "kube-api-access-768pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:18.465165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.465130 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn" (OuterVolumeSpecName: "kube-api-access-nrrjn") pod "be74a39c-bfdc-44d2-8a14-ce8094bba24a" (UID: "be74a39c-bfdc-44d2-8a14-ce8094bba24a"). InnerVolumeSpecName "kube-api-access-nrrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:18.563785 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.563712 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrjn\" (UniqueName: \"kubernetes.io/projected/be74a39c-bfdc-44d2-8a14-ce8094bba24a-kube-api-access-nrrjn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:18.563967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.563797 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a897a7f3-f982-47b0-a531-bc87a2b09dad-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:18.563967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.563813 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be74a39c-bfdc-44d2-8a14-ce8094bba24a-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:18.563967 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.563826 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768pc\" (UniqueName: \"kubernetes.io/projected/a897a7f3-f982-47b0-a531-bc87a2b09dad-kube-api-access-768pc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:18.578420 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.578388 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:18.628107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.628009 2581 generic.go:356] "Generic (PLEG): container finished" podID="a897a7f3-f982-47b0-a531-bc87a2b09dad" containerID="616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0" exitCode=0 Sep 10 15:16:18.628107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.628061 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" Sep 10 15:16:18.628107 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.628098 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" event={"ID":"a897a7f3-f982-47b0-a531-bc87a2b09dad","Type":"ContainerDied","Data":"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0"} Sep 10 15:16:18.628389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.628155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk" event={"ID":"a897a7f3-f982-47b0-a531-bc87a2b09dad","Type":"ContainerDied","Data":"1f644ace1075e5ed9928c7cef2eab27ccde8aaf8e074d780ec2cb7377f784e5c"} Sep 10 15:16:18.628389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.628184 2581 scope.go:117] "RemoveContainer" containerID="616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0" Sep 10 15:16:18.629497 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.629469 2581 generic.go:356] "Generic (PLEG): container finished" podID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" containerID="d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d" exitCode=0 Sep 10 15:16:18.629621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.629518 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" Sep 10 15:16:18.629621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.629552 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" event={"ID":"be74a39c-bfdc-44d2-8a14-ce8094bba24a","Type":"ContainerDied","Data":"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d"} Sep 10 15:16:18.629621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.629599 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs" event={"ID":"be74a39c-bfdc-44d2-8a14-ce8094bba24a","Type":"ContainerDied","Data":"37e8c4ae9d83003abe8715b4880b9c059f0e33859c326dce8680d6ab0653f9db"} Sep 10 15:16:18.639551 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.639532 2581 scope.go:117] "RemoveContainer" containerID="616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0" Sep 10 15:16:18.639912 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:16:18.639886 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0\": container with ID starting with 616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0 not found: ID does not exist" containerID="616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0" Sep 10 15:16:18.640007 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.639918 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0"} err="failed to get container status \"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0\": rpc error: code = NotFound desc = could not find container \"616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0\": container with ID starting with 616e40afe3bca26409dd0ed32607ca2acd9a9dd2bba2181dc86afc55c72d69b0 not found: ID does not exist" Sep 10 15:16:18.640007 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.639934 2581 scope.go:117] "RemoveContainer" containerID="d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d" Sep 10 15:16:18.648541 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.648521 2581 scope.go:117] "RemoveContainer" containerID="d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d" Sep 10 15:16:18.648836 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:16:18.648815 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d\": container with ID starting with d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d not found: ID does not exist" containerID="d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d" Sep 10 15:16:18.648890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.648843 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d"} err="failed to get container status \"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d\": rpc error: code = NotFound desc = could not find container \"d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d\": container with ID starting with d99d437d7676f59ad268878b3f1e68e1f898596ac00daace0763ec52ca58ab6d not found: ID does not exist" Sep 10 15:16:18.654198 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.654172 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:16:18.659442 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.659418 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-06cc0e-5bd44fc8c7-tvhqs"] Sep 10 15:16:18.673250 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.673223 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:16:18.677338 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:18.677313 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-06cc0e-6b9d795f7d-f4gfk"] Sep 10 15:16:20.230614 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:20.230576 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a897a7f3-f982-47b0-a531-bc87a2b09dad" path="/var/lib/kubelet/pods/a897a7f3-f982-47b0-a531-bc87a2b09dad/volumes" Sep 10 15:16:20.231033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:20.230976 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" path="/var/lib/kubelet/pods/be74a39c-bfdc-44d2-8a14-ce8094bba24a/volumes" Sep 10 15:16:20.231253 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:20.231240 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03f8df6-132e-4d14-b9e9-009a03e33941" path="/var/lib/kubelet/pods/c03f8df6-132e-4d14-b9e9-009a03e33941/volumes" Sep 10 15:16:24.320440 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:24.320410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:24.386165 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:24.386127 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:24.593927 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:24.593825 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:28.051769 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.051710 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx"] Sep 10 15:16:28.052184 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052171 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f8df6-132e-4d14-b9e9-009a03e33941" containerName="revoke-keys" Sep 10 15:16:28.052184 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052185 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f8df6-132e-4d14-b9e9-009a03e33941" containerName="revoke-keys" Sep 10 15:16:28.052262 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052197 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" containerName="payload-processing" Sep 10 15:16:28.052262 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052203 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" containerName="payload-processing" Sep 10 15:16:28.052262 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052214 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a897a7f3-f982-47b0-a531-bc87a2b09dad" containerName="payload-pre-processing" Sep 10 15:16:28.052262 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052220 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a897a7f3-f982-47b0-a531-bc87a2b09dad" containerName="payload-pre-processing" Sep 10 15:16:28.052377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052290 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a897a7f3-f982-47b0-a531-bc87a2b09dad" containerName="payload-pre-processing" Sep 10 15:16:28.052377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052298 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03f8df6-132e-4d14-b9e9-009a03e33941" containerName="revoke-keys" Sep 10 15:16:28.052377 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.052305 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be74a39c-bfdc-44d2-8a14-ce8094bba24a" containerName="payload-processing" Sep 10 15:16:28.056502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.056484 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:28.058830 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.058808 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:16:28.063169 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.063139 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx"] Sep 10 15:16:28.151839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.151805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx\" (UID: \"1acfa31a-9c66-4d5a-bb35-e82cb61608ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:28.181279 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.181227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:28.253328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.253295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx\" (UID: \"1acfa31a-9c66-4d5a-bb35-e82cb61608ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:28.254154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.254128 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx\" (UID: \"1acfa31a-9c66-4d5a-bb35-e82cb61608ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:28.370615 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.370523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:28.646010 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.645980 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx"] Sep 10 15:16:28.648171 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:16:28.648138 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acfa31a_9c66_4d5a_bb35_e82cb61608ea.slice/crio-b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b WatchSource:0}: Error finding container b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b: Status 404 returned error can't find the container with id b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b Sep 10 15:16:28.688833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:28.688796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" event={"ID":"1acfa31a-9c66-4d5a-bb35-e82cb61608ea","Type":"ContainerStarted","Data":"b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b"} Sep 10 15:16:29.696672 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:29.696577 2581 generic.go:356] "Generic (PLEG): container finished" podID="1acfa31a-9c66-4d5a-bb35-e82cb61608ea" containerID="195f60150a25663f1355d1b5b0d7298bcb9c1c44611b0a56f0608aed0cc40673" exitCode=0 Sep 10 15:16:29.696672 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:29.696638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" event={"ID":"1acfa31a-9c66-4d5a-bb35-e82cb61608ea","Type":"ContainerDied","Data":"195f60150a25663f1355d1b5b0d7298bcb9c1c44611b0a56f0608aed0cc40673"} Sep 10 15:16:30.848079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:30.848053 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:30.877505 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:30.877468 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca\") pod \"1acfa31a-9c66-4d5a-bb35-e82cb61608ea\" (UID: \"1acfa31a-9c66-4d5a-bb35-e82cb61608ea\") " Sep 10 15:16:30.877835 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:30.877814 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1acfa31a-9c66-4d5a-bb35-e82cb61608ea" (UID: "1acfa31a-9c66-4d5a-bb35-e82cb61608ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:30.978434 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:30.978341 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1acfa31a-9c66-4d5a-bb35-e82cb61608ea-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:31.709649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:31.709613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" event={"ID":"1acfa31a-9c66-4d5a-bb35-e82cb61608ea","Type":"ContainerDied","Data":"b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b"} Sep 10 15:16:31.709649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:31.709646 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx" Sep 10 15:16:31.709887 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:31.709690 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9fbeb3e79fbc999a0e7704e2a8b56bdceceda69680ef548a21bb4eb80679d4b" Sep 10 15:16:43.567421 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.567386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:43.589789 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.589736 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:16:43.590032 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.590008 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" podUID="e6e6efd4-4c86-4065-8dba-3db82d0da337" containerName="payload-processing" containerID="cri-o://753b47d16b8c98bb37c4087904955c60b8ed768b35228ffe61c76b51ef4052e7" gracePeriod=30 Sep 10 15:16:43.601714 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.601683 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:16:43.602181 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.601967 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" podUID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" containerName="payload-pre-processing" containerID="cri-o://af165bc78c9ec3f2ffd2d25f9f1cf4d03ef955046d521a8591ca381deb6146f2" gracePeriod=30 Sep 10 15:16:43.633237 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.633206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:43.702665 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.702631 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:43.760451 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.760419 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx"] Sep 10 15:16:43.763859 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.763829 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-13ee06-k9wfx"] Sep 10 15:16:43.767854 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.767781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:43.783867 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.783831 2581 generic.go:356] "Generic (PLEG): container finished" podID="e6e6efd4-4c86-4065-8dba-3db82d0da337" containerID="753b47d16b8c98bb37c4087904955c60b8ed768b35228ffe61c76b51ef4052e7" exitCode=0 Sep 10 15:16:43.784037 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.783922 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" event={"ID":"e6e6efd4-4c86-4065-8dba-3db82d0da337","Type":"ContainerDied","Data":"753b47d16b8c98bb37c4087904955c60b8ed768b35228ffe61c76b51ef4052e7"} Sep 10 15:16:43.785944 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.785916 2581 generic.go:356] "Generic (PLEG): container finished" podID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" containerID="af165bc78c9ec3f2ffd2d25f9f1cf4d03ef955046d521a8591ca381deb6146f2" exitCode=0 Sep 10 15:16:43.786050 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.785980 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" event={"ID":"c2df2bcc-a874-4a8e-9b0f-0d329df610de","Type":"ContainerDied","Data":"af165bc78c9ec3f2ffd2d25f9f1cf4d03ef955046d521a8591ca381deb6146f2"} Sep 10 15:16:43.901545 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.901515 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:16:43.908943 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.908917 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:16:43.997020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.996973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume\") pod \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " Sep 10 15:16:43.997020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.997048 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ds6l\" (UniqueName: \"kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l\") pod \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\" (UID: \"c2df2bcc-a874-4a8e-9b0f-0d329df610de\") " Sep 10 15:16:43.997296 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.997069 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume\") pod \"e6e6efd4-4c86-4065-8dba-3db82d0da337\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " Sep 10 15:16:43.997296 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.997140 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgm9p\" (UniqueName: \"kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p\") pod \"e6e6efd4-4c86-4065-8dba-3db82d0da337\" (UID: \"e6e6efd4-4c86-4065-8dba-3db82d0da337\") " Sep 10 15:16:43.997520 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.997490 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6e6efd4-4c86-4065-8dba-3db82d0da337" (UID: "e6e6efd4-4c86-4065-8dba-3db82d0da337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:43.997798 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.997745 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2df2bcc-a874-4a8e-9b0f-0d329df610de" (UID: "c2df2bcc-a874-4a8e-9b0f-0d329df610de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:43.999519 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.999488 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l" (OuterVolumeSpecName: "kube-api-access-2ds6l") pod "c2df2bcc-a874-4a8e-9b0f-0d329df610de" (UID: "c2df2bcc-a874-4a8e-9b0f-0d329df610de"). InnerVolumeSpecName "kube-api-access-2ds6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:43.999614 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:43.999528 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p" (OuterVolumeSpecName: "kube-api-access-xgm9p") pod "e6e6efd4-4c86-4065-8dba-3db82d0da337" (UID: "e6e6efd4-4c86-4065-8dba-3db82d0da337"). InnerVolumeSpecName "kube-api-access-xgm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:44.098168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.098129 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgm9p\" (UniqueName: \"kubernetes.io/projected/e6e6efd4-4c86-4065-8dba-3db82d0da337-kube-api-access-xgm9p\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.098168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.098158 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2df2bcc-a874-4a8e-9b0f-0d329df610de-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.098168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.098168 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ds6l\" (UniqueName: \"kubernetes.io/projected/c2df2bcc-a874-4a8e-9b0f-0d329df610de-kube-api-access-2ds6l\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.098168 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.098178 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e6efd4-4c86-4065-8dba-3db82d0da337-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.167361 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.167271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:44.232795 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.231838 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acfa31a-9c66-4d5a-bb35-e82cb61608ea" path="/var/lib/kubelet/pods/1acfa31a-9c66-4d5a-bb35-e82cb61608ea/volumes" Sep 10 15:16:44.796194 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.796152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" event={"ID":"e6e6efd4-4c86-4065-8dba-3db82d0da337","Type":"ContainerDied","Data":"3fa739db1acf76975a914fffff4ca7dfb0e4c8027d0164e8cf3b9da14987c4e0"} Sep 10 15:16:44.796655 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.796218 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r" Sep 10 15:16:44.796655 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.796256 2581 scope.go:117] "RemoveContainer" containerID="753b47d16b8c98bb37c4087904955c60b8ed768b35228ffe61c76b51ef4052e7" Sep 10 15:16:44.798502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.798404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" event={"ID":"c2df2bcc-a874-4a8e-9b0f-0d329df610de","Type":"ContainerDied","Data":"2358426c3abe379509ee4f57cbb95f694b839b26438cc8ec1ebcf34be17966b8"} Sep 10 15:16:44.798607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.798515 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95" Sep 10 15:16:44.807304 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.807282 2581 scope.go:117] "RemoveContainer" containerID="af165bc78c9ec3f2ffd2d25f9f1cf4d03ef955046d521a8591ca381deb6146f2" Sep 10 15:16:44.815551 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.815523 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:16:44.819456 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.819434 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13ee06-69c5df7898-4kg95"] Sep 10 15:16:44.829242 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.829219 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:16:44.831059 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:44.831038 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13ee06-6cd9887965-m4d7r"] Sep 10 15:16:46.230614 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:46.230574 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" path="/var/lib/kubelet/pods/c2df2bcc-a874-4a8e-9b0f-0d329df610de/volumes" Sep 10 15:16:46.231026 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:46.230963 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e6efd4-4c86-4065-8dba-3db82d0da337" path="/var/lib/kubelet/pods/e6e6efd4-4c86-4065-8dba-3db82d0da337/volumes" Sep 10 15:16:49.152480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:49.152448 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:16:52.185559 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:16:52.185520 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:20.565429 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:20.565347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:25.443237 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:25.443203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:26.212009 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:26.211973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:26.312091 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:26.312064 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:17:26.315910 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:26.315877 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:17:31.745445 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:31.745404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:31.812302 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:31.812263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:32.005955 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.005838 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:32.339311 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339217 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t"] Sep 10 15:17:32.339675 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339661 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e6efd4-4c86-4065-8dba-3db82d0da337" containerName="payload-processing" Sep 10 15:17:32.339718 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339678 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e6efd4-4c86-4065-8dba-3db82d0da337" containerName="payload-processing" Sep 10 15:17:32.339718 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339710 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" containerName="payload-pre-processing" Sep 10 15:17:32.339718 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339716 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" containerName="payload-pre-processing" Sep 10 15:17:32.339846 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339727 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acfa31a-9c66-4d5a-bb35-e82cb61608ea" containerName="revoke-keys" Sep 10 15:17:32.339846 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339741 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acfa31a-9c66-4d5a-bb35-e82cb61608ea" containerName="revoke-keys" Sep 10 15:17:32.339846 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339821 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e6efd4-4c86-4065-8dba-3db82d0da337" containerName="payload-processing" Sep 10 15:17:32.339846 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339832 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2df2bcc-a874-4a8e-9b0f-0d329df610de" containerName="payload-pre-processing" Sep 10 15:17:32.339846 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.339838 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acfa31a-9c66-4d5a-bb35-e82cb61608ea" containerName="revoke-keys" Sep 10 15:17:32.343162 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.343142 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:32.346015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.345989 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:17:32.352697 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.352664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t"] Sep 10 15:17:32.431940 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.431896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t\" (UID: \"dd6c9166-9381-46dc-87ad-1044cdff788f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:32.533288 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.533239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t\" (UID: \"dd6c9166-9381-46dc-87ad-1044cdff788f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:32.533923 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.533901 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t\" (UID: \"dd6c9166-9381-46dc-87ad-1044cdff788f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:32.605596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.605502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:32.655588 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.655550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:32.929676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:32.929622 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t"] Sep 10 15:17:32.932151 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:17:32.932125 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c9166_9381_46dc_87ad_1044cdff788f.slice/crio-c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8 WatchSource:0}: Error finding container c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8: Status 404 returned error can't find the container with id c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8 Sep 10 15:17:33.077689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:33.077653 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" event={"ID":"dd6c9166-9381-46dc-87ad-1044cdff788f","Type":"ContainerStarted","Data":"c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8"} Sep 10 15:17:34.085256 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:34.085219 2581 generic.go:356] "Generic (PLEG): container finished" podID="dd6c9166-9381-46dc-87ad-1044cdff788f" containerID="672289ac9491f7b08e4e7584cf12c6635af11ca2fffcaca004381ba3c77ef795" exitCode=0 Sep 10 15:17:34.085626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:34.085262 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" event={"ID":"dd6c9166-9381-46dc-87ad-1044cdff788f","Type":"ContainerDied","Data":"672289ac9491f7b08e4e7584cf12c6635af11ca2fffcaca004381ba3c77ef795"} Sep 10 15:17:35.234515 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:35.234489 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:35.354807 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:35.354674 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca\") pod \"dd6c9166-9381-46dc-87ad-1044cdff788f\" (UID: \"dd6c9166-9381-46dc-87ad-1044cdff788f\") " Sep 10 15:17:35.355080 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:35.355053 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd6c9166-9381-46dc-87ad-1044cdff788f" (UID: "dd6c9166-9381-46dc-87ad-1044cdff788f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:35.456146 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:35.456102 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c9166-9381-46dc-87ad-1044cdff788f-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:36.098048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:36.098012 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" Sep 10 15:17:36.098048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:36.098045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t" event={"ID":"dd6c9166-9381-46dc-87ad-1044cdff788f","Type":"ContainerDied","Data":"c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8"} Sep 10 15:17:36.098272 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:36.098094 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7436cb57554b8d120e3ddf5417aebe404cf1ff6452f198f4794e82522bf58f8" Sep 10 15:17:42.954961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.954922 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd"] Sep 10 15:17:42.957353 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.957326 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6333de-29817555-7jjgd"] Sep 10 15:17:42.979198 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.979165 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:17:42.979433 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.979405 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" podUID="624b0bd0-2a1b-44fd-803f-563438287437" containerName="payload-processing" containerID="cri-o://d53a6ea045e818644caa4204b24dc347d8bbda046e592cedc251fd73fe5c2824" gracePeriod=30 Sep 10 15:17:42.989208 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.989178 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:17:42.989961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:42.989930 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" podUID="795b45ba-6a7d-4481-a08e-ba50f5c76514" containerName="payload-pre-processing" containerID="cri-o://53883411940c1994e9e4ef407165d9324bd2c63b8b99110ad701ae3863f36a12" gracePeriod=30 Sep 10 15:17:43.026563 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.026531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:43.127816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.127517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:43.140961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.140725 2581 generic.go:356] "Generic (PLEG): container finished" podID="624b0bd0-2a1b-44fd-803f-563438287437" containerID="d53a6ea045e818644caa4204b24dc347d8bbda046e592cedc251fd73fe5c2824" exitCode=0 Sep 10 15:17:43.140961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.140886 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" event={"ID":"624b0bd0-2a1b-44fd-803f-563438287437","Type":"ContainerDied","Data":"d53a6ea045e818644caa4204b24dc347d8bbda046e592cedc251fd73fe5c2824"} Sep 10 15:17:43.143395 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.143360 2581 generic.go:356] "Generic (PLEG): container finished" podID="795b45ba-6a7d-4481-a08e-ba50f5c76514" containerID="53883411940c1994e9e4ef407165d9324bd2c63b8b99110ad701ae3863f36a12" exitCode=0 Sep 10 15:17:43.143537 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.143454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" event={"ID":"795b45ba-6a7d-4481-a08e-ba50f5c76514","Type":"ContainerDied","Data":"53883411940c1994e9e4ef407165d9324bd2c63b8b99110ad701ae3863f36a12"} Sep 10 15:17:43.149278 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.149230 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t"] Sep 10 15:17:43.153171 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.153125 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6333de-zvd8t"] Sep 10 15:17:43.231984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.231935 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:43.267465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.267441 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:17:43.290092 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.290069 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:17:43.329397 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.329368 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume\") pod \"624b0bd0-2a1b-44fd-803f-563438287437\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " Sep 10 15:17:43.329567 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.329442 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmx2z\" (UniqueName: \"kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z\") pod \"624b0bd0-2a1b-44fd-803f-563438287437\" (UID: \"624b0bd0-2a1b-44fd-803f-563438287437\") " Sep 10 15:17:43.329818 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.329787 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "624b0bd0-2a1b-44fd-803f-563438287437" (UID: "624b0bd0-2a1b-44fd-803f-563438287437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:43.331663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.331645 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z" (OuterVolumeSpecName: "kube-api-access-cmx2z") pod "624b0bd0-2a1b-44fd-803f-563438287437" (UID: "624b0bd0-2a1b-44fd-803f-563438287437"). InnerVolumeSpecName "kube-api-access-cmx2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:43.430608 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.430573 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm54r\" (UniqueName: \"kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r\") pod \"795b45ba-6a7d-4481-a08e-ba50f5c76514\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " Sep 10 15:17:43.430834 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.430694 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume\") pod \"795b45ba-6a7d-4481-a08e-ba50f5c76514\" (UID: \"795b45ba-6a7d-4481-a08e-ba50f5c76514\") " Sep 10 15:17:43.430961 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.430942 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624b0bd0-2a1b-44fd-803f-563438287437-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:43.431008 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.430968 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmx2z\" (UniqueName: \"kubernetes.io/projected/624b0bd0-2a1b-44fd-803f-563438287437-kube-api-access-cmx2z\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:43.431074 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.431050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "795b45ba-6a7d-4481-a08e-ba50f5c76514" (UID: "795b45ba-6a7d-4481-a08e-ba50f5c76514"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:43.433074 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.433051 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r" (OuterVolumeSpecName: "kube-api-access-vm54r") pod "795b45ba-6a7d-4481-a08e-ba50f5c76514" (UID: "795b45ba-6a7d-4481-a08e-ba50f5c76514"). InnerVolumeSpecName "kube-api-access-vm54r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:43.531429 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.531387 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795b45ba-6a7d-4481-a08e-ba50f5c76514-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:43.531429 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.531420 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm54r\" (UniqueName: \"kubernetes.io/projected/795b45ba-6a7d-4481-a08e-ba50f5c76514-kube-api-access-vm54r\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:43.550277 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:43.550214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:44.150626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.150596 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" Sep 10 15:17:44.151154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.150642 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw" event={"ID":"795b45ba-6a7d-4481-a08e-ba50f5c76514","Type":"ContainerDied","Data":"bd819b7fd7cdb1c3e3ae41485b00c72166a9d60b67ef0f8ae976468e6dab1eb0"} Sep 10 15:17:44.151154 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.150710 2581 scope.go:117] "RemoveContainer" containerID="53883411940c1994e9e4ef407165d9324bd2c63b8b99110ad701ae3863f36a12" Sep 10 15:17:44.152461 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.152434 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" event={"ID":"624b0bd0-2a1b-44fd-803f-563438287437","Type":"ContainerDied","Data":"419cdc51e59b675aab38d42d09ef36c9f2ed7907e37226fe21d11b4e3986fb1b"} Sep 10 15:17:44.152564 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.152463 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2" Sep 10 15:17:44.164419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.164284 2581 scope.go:117] "RemoveContainer" containerID="d53a6ea045e818644caa4204b24dc347d8bbda046e592cedc251fd73fe5c2824" Sep 10 15:17:44.181922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.181893 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:17:44.186361 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.186335 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6333de-5b7b66d766-x8ksw"] Sep 10 15:17:44.198037 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.197967 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:17:44.200365 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.200344 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6333de-79c4d76547-trvl2"] Sep 10 15:17:44.232248 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.232216 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624b0bd0-2a1b-44fd-803f-563438287437" path="/var/lib/kubelet/pods/624b0bd0-2a1b-44fd-803f-563438287437/volumes" Sep 10 15:17:44.232562 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.232550 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795b45ba-6a7d-4481-a08e-ba50f5c76514" path="/var/lib/kubelet/pods/795b45ba-6a7d-4481-a08e-ba50f5c76514/volumes" Sep 10 15:17:44.232827 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.232814 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91304ed2-1a53-4f96-af5a-81bc51d52ce9" path="/var/lib/kubelet/pods/91304ed2-1a53-4f96-af5a-81bc51d52ce9/volumes" Sep 10 15:17:44.233072 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:44.233060 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd6c9166-9381-46dc-87ad-1044cdff788f" path="/var/lib/kubelet/pods/dd6c9166-9381-46dc-87ad-1044cdff788f/volumes" Sep 10 15:17:49.450822 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:49.450790 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:49.551369 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:49.551332 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:49.721015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:49.720928 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:50.068095 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068054 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz"] Sep 10 15:17:50.068539 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068520 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624b0bd0-2a1b-44fd-803f-563438287437" containerName="payload-processing" Sep 10 15:17:50.068627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068542 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="624b0bd0-2a1b-44fd-803f-563438287437" containerName="payload-processing" Sep 10 15:17:50.068627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068572 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795b45ba-6a7d-4481-a08e-ba50f5c76514" containerName="payload-pre-processing" Sep 10 15:17:50.068627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068581 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="795b45ba-6a7d-4481-a08e-ba50f5c76514" containerName="payload-pre-processing" Sep 10 15:17:50.068627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068592 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6c9166-9381-46dc-87ad-1044cdff788f" containerName="revoke-keys" Sep 10 15:17:50.068627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068600 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6c9166-9381-46dc-87ad-1044cdff788f" containerName="revoke-keys" Sep 10 15:17:50.068908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068711 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="624b0bd0-2a1b-44fd-803f-563438287437" containerName="payload-processing" Sep 10 15:17:50.068908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068724 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="795b45ba-6a7d-4481-a08e-ba50f5c76514" containerName="payload-pre-processing" Sep 10 15:17:50.068908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.068736 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6c9166-9381-46dc-87ad-1044cdff788f" containerName="revoke-keys" Sep 10 15:17:50.073232 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.073209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:50.078305 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.078273 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\"" Sep 10 15:17:50.081433 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.081397 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz"] Sep 10 15:17:50.190285 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.190249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz\" (UID: \"09a4273e-534e-4351-a016-b1199481fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:50.230703 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.230666 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:17:50.291400 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.291364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz\" (UID: \"09a4273e-534e-4351-a016-b1199481fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:50.292042 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.292023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz\" (UID: \"09a4273e-534e-4351-a016-b1199481fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:50.386278 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.386175 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:50.658656 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:50.658632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz"] Sep 10 15:17:50.660438 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:17:50.660408 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a4273e_534e_4351_a016_b1199481fbf9.slice/crio-1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573 WatchSource:0}: Error finding container 1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573: Status 404 returned error can't find the container with id 1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573 Sep 10 15:17:51.196922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:51.196806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" event={"ID":"09a4273e-534e-4351-a016-b1199481fbf9","Type":"ContainerStarted","Data":"b81b45813549f6e16ae2ba757180f48a89606cd828e0f83af3c32af054788129"} Sep 10 15:17:51.196922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:51.196875 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" event={"ID":"09a4273e-534e-4351-a016-b1199481fbf9","Type":"ContainerStarted","Data":"1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573"} Sep 10 15:17:51.212788 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:51.212722 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" podStartSLOduration=1.212705725 podStartE2EDuration="1.212705725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:51.211300172 +0000 UTC m=+1825.505602300" watchObservedRunningTime="2026-09-10 15:17:51.212705725 +0000 UTC m=+1825.507007854" Sep 10 15:17:52.205088 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:52.205041 2581 generic.go:356] "Generic (PLEG): container finished" podID="09a4273e-534e-4351-a016-b1199481fbf9" containerID="b81b45813549f6e16ae2ba757180f48a89606cd828e0f83af3c32af054788129" exitCode=0 Sep 10 15:17:52.205521 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:52.205119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" event={"ID":"09a4273e-534e-4351-a016-b1199481fbf9","Type":"ContainerDied","Data":"b81b45813549f6e16ae2ba757180f48a89606cd828e0f83af3c32af054788129"} Sep 10 15:17:53.358907 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:53.358880 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:53.416890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:53.416857 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca\") pod \"09a4273e-534e-4351-a016-b1199481fbf9\" (UID: \"09a4273e-534e-4351-a016-b1199481fbf9\") " Sep 10 15:17:53.417212 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:53.417187 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09a4273e-534e-4351-a016-b1199481fbf9" (UID: "09a4273e-534e-4351-a016-b1199481fbf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:53.517590 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:53.517556 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a4273e-534e-4351-a016-b1199481fbf9-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:17:54.218158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:54.218132 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" Sep 10 15:17:54.218158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:54.218149 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz" event={"ID":"09a4273e-534e-4351-a016-b1199481fbf9","Type":"ContainerDied","Data":"1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573"} Sep 10 15:17:54.218371 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:17:54.218191 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1755593f605f4770184bf8b5dc8b8b47efe14628ab0a04d51e21999d883b7573" Sep 10 15:18:00.515496 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.515460 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb"] Sep 10 15:18:00.518498 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.518474 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-decc52-29817555-7wkxb"] Sep 10 15:18:00.521334 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.521295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:00.539596 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.539549 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:18:00.539890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.539862 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" podUID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" containerName="payload-processing" containerID="cri-o://c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1" gracePeriod=30 Sep 10 15:18:00.549964 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.549934 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:18:00.550193 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.550153 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" podUID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" containerName="payload-pre-processing" containerID="cri-o://f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4" gracePeriod=30 Sep 10 15:18:00.587457 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.587395 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:00.653663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.653627 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:00.709712 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.709677 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz"] Sep 10 15:18:00.713097 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.713044 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-decc52-hdqvz"] Sep 10 15:18:00.754431 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.754328 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:00.827831 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.827804 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:18:00.830882 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.830861 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:18:00.988033 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.987979 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume\") pod \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " Sep 10 15:18:00.988244 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.988104 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zhhr\" (UniqueName: \"kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr\") pod \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\" (UID: \"d8953ab1-88fc-4a78-8d63-5d4ef64abc87\") " Sep 10 15:18:00.988244 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.988123 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume\") pod \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " Sep 10 15:18:00.988244 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.988167 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbv5\" (UniqueName: \"kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5\") pod \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\" (UID: \"3482ff74-63e4-401d-80c5-4b4c1ea224a1\") " Sep 10 15:18:00.988441 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.988398 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8953ab1-88fc-4a78-8d63-5d4ef64abc87" (UID: "d8953ab1-88fc-4a78-8d63-5d4ef64abc87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:00.988502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.988478 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3482ff74-63e4-401d-80c5-4b4c1ea224a1" (UID: "3482ff74-63e4-401d-80c5-4b4c1ea224a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:00.990479 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.990447 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr" (OuterVolumeSpecName: "kube-api-access-7zhhr") pod "d8953ab1-88fc-4a78-8d63-5d4ef64abc87" (UID: "d8953ab1-88fc-4a78-8d63-5d4ef64abc87"). InnerVolumeSpecName "kube-api-access-7zhhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:00.990593 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:00.990509 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5" (OuterVolumeSpecName: "kube-api-access-gzbv5") pod "3482ff74-63e4-401d-80c5-4b4c1ea224a1" (UID: "3482ff74-63e4-401d-80c5-4b4c1ea224a1"). InnerVolumeSpecName "kube-api-access-gzbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:01.075452 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.075360 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:01.089437 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.089408 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhhr\" (UniqueName: \"kubernetes.io/projected/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-kube-api-access-7zhhr\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:01.089437 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.089438 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3482ff74-63e4-401d-80c5-4b4c1ea224a1-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:01.089581 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.089449 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbv5\" (UniqueName: \"kubernetes.io/projected/3482ff74-63e4-401d-80c5-4b4c1ea224a1-kube-api-access-gzbv5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:01.089581 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.089459 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8953ab1-88fc-4a78-8d63-5d4ef64abc87-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:01.260735 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.260700 2581 generic.go:356] "Generic (PLEG): container finished" podID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" containerID="c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1" exitCode=0 Sep 10 15:18:01.260987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.260748 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" Sep 10 15:18:01.260987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.260786 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" event={"ID":"d8953ab1-88fc-4a78-8d63-5d4ef64abc87","Type":"ContainerDied","Data":"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1"} Sep 10 15:18:01.260987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.260836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5" event={"ID":"d8953ab1-88fc-4a78-8d63-5d4ef64abc87","Type":"ContainerDied","Data":"2b1f69fd197a7482bfa0db3caa2e47fc21be88aacd8b47de053908dd644b2625"} Sep 10 15:18:01.260987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.260866 2581 scope.go:117] "RemoveContainer" containerID="c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1" Sep 10 15:18:01.262284 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.262260 2581 generic.go:356] "Generic (PLEG): container finished" podID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" containerID="f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4" exitCode=0 Sep 10 15:18:01.262399 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.262304 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" Sep 10 15:18:01.262399 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.262347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" event={"ID":"3482ff74-63e4-401d-80c5-4b4c1ea224a1","Type":"ContainerDied","Data":"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4"} Sep 10 15:18:01.262399 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.262391 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7" event={"ID":"3482ff74-63e4-401d-80c5-4b4c1ea224a1","Type":"ContainerDied","Data":"475cdafb813ed83935e66531d1a6e195b8201193b03164ec6a7f3483fdc32b81"} Sep 10 15:18:01.270901 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.270873 2581 scope.go:117] "RemoveContainer" containerID="c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1" Sep 10 15:18:01.271294 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:18:01.271271 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1\": container with ID starting with c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1 not found: ID does not exist" containerID="c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1" Sep 10 15:18:01.271401 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.271299 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1"} err="failed to get container status \"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1\": rpc error: code = NotFound desc = could not find container \"c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1\": container with ID starting with c49f7658b1a4a67ec7b0f210c54afcaf0ece7439659d1d1e0b4b0b7447726fa1 not found: ID does not exist" Sep 10 15:18:01.271401 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.271320 2581 scope.go:117] "RemoveContainer" containerID="f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4" Sep 10 15:18:01.280221 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.280199 2581 scope.go:117] "RemoveContainer" containerID="f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4" Sep 10 15:18:01.280515 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:18:01.280499 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4\": container with ID starting with f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4 not found: ID does not exist" containerID="f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4" Sep 10 15:18:01.280566 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.280526 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4"} err="failed to get container status \"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4\": rpc error: code = NotFound desc = could not find container \"f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4\": container with ID starting with f01456b9ee278cba2841daf9303ae24d69143a77df2051bc6efa674c9ce8f4a4 not found: ID does not exist" Sep 10 15:18:01.284981 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.284953 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:18:01.288588 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.288565 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-decc52-79f8dd794b-482b7"] Sep 10 15:18:01.298337 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.298313 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:18:01.301833 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:01.301811 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-decc52-69fdccc87-2t5d5"] Sep 10 15:18:02.229652 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:02.229620 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a4273e-534e-4351-a016-b1199481fbf9" path="/var/lib/kubelet/pods/09a4273e-534e-4351-a016-b1199481fbf9/volumes" Sep 10 15:18:02.230052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:02.229998 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" path="/var/lib/kubelet/pods/3482ff74-63e4-401d-80c5-4b4c1ea224a1/volumes" Sep 10 15:18:02.230235 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:02.230222 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04f12ee-0041-4239-86a4-1371639e0573" path="/var/lib/kubelet/pods/a04f12ee-0041-4239-86a4-1371639e0573/volumes" Sep 10 15:18:02.230502 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:02.230490 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" path="/var/lib/kubelet/pods/d8953ab1-88fc-4a78-8d63-5d4ef64abc87/volumes" Sep 10 15:18:06.967548 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:06.967498 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:07.034412 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:07.034380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:07.219729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:07.219640 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:07.778464 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:07.778397 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:14.463588 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:14.463500 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:17.680953 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:17.680904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:18.252347 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.252313 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:18.253470 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.253440 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms"] Sep 10 15:18:18.256037 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.256016 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d5a2df-29817555-wqzms"] Sep 10 15:18:18.279568 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.279534 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:18:18.279872 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.279839 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" podUID="033b206f-18ee-4703-bc44-266e9ce196d9" containerName="payload-processing" containerID="cri-o://d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5" gracePeriod=30 Sep 10 15:18:18.286079 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.286043 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:18:18.286269 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.286244 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" podUID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" containerName="payload-pre-processing" containerID="cri-o://c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0" gracePeriod=30 Sep 10 15:18:18.351327 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.351295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:18.449973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.449882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:18.551459 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.551420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:18.567169 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.567142 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:18:18.569817 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.569797 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:18:18.642618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.642551 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lm7r\" (UniqueName: \"kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r\") pod \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " Sep 10 15:18:18.642618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.642589 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume\") pod \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\" (UID: \"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa\") " Sep 10 15:18:18.642618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.642609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zj74\" (UniqueName: \"kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74\") pod \"033b206f-18ee-4703-bc44-266e9ce196d9\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " Sep 10 15:18:18.642618 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.642634 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume\") pod \"033b206f-18ee-4703-bc44-266e9ce196d9\" (UID: \"033b206f-18ee-4703-bc44-266e9ce196d9\") " Sep 10 15:18:18.643020 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.642982 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" (UID: "2b5b0cca-6c3c-49f5-a9c9-510470ba29aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:18.643074 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.643041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "033b206f-18ee-4703-bc44-266e9ce196d9" (UID: "033b206f-18ee-4703-bc44-266e9ce196d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:18.645093 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.645064 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74" (OuterVolumeSpecName: "kube-api-access-2zj74") pod "033b206f-18ee-4703-bc44-266e9ce196d9" (UID: "033b206f-18ee-4703-bc44-266e9ce196d9"). InnerVolumeSpecName "kube-api-access-2zj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:18.645201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.645101 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r" (OuterVolumeSpecName: "kube-api-access-4lm7r") pod "2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" (UID: "2b5b0cca-6c3c-49f5-a9c9-510470ba29aa"). InnerVolumeSpecName "kube-api-access-4lm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:18.743475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.743434 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lm7r\" (UniqueName: \"kubernetes.io/projected/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-kube-api-access-4lm7r\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.743475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.743469 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.743475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.743478 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj74\" (UniqueName: \"kubernetes.io/projected/033b206f-18ee-4703-bc44-266e9ce196d9-kube-api-access-2zj74\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.743475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.743487 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033b206f-18ee-4703-bc44-266e9ce196d9-plugins-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.829173 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:18.829083 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:19.372535 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.372500 2581 generic.go:356] "Generic (PLEG): container finished" podID="033b206f-18ee-4703-bc44-266e9ce196d9" containerID="d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5" exitCode=0 Sep 10 15:18:19.372730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.372548 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" Sep 10 15:18:19.372730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.372571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" event={"ID":"033b206f-18ee-4703-bc44-266e9ce196d9","Type":"ContainerDied","Data":"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5"} Sep 10 15:18:19.372730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.372621 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s" event={"ID":"033b206f-18ee-4703-bc44-266e9ce196d9","Type":"ContainerDied","Data":"f027f82f3bca26f378a402ddc94ed534b1f2c2a67c154b996fd2d2314ec35dbc"} Sep 10 15:18:19.372730 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.372650 2581 scope.go:117] "RemoveContainer" containerID="d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5" Sep 10 15:18:19.374158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.374132 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" containerID="c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0" exitCode=0 Sep 10 15:18:19.374239 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.374187 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" Sep 10 15:18:19.374304 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.374187 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" event={"ID":"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa","Type":"ContainerDied","Data":"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0"} Sep 10 15:18:19.374353 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.374317 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9" event={"ID":"2b5b0cca-6c3c-49f5-a9c9-510470ba29aa","Type":"ContainerDied","Data":"714fd80aef52f2edb33ca223b7481aba77ecc507e22f3ed13a9dc17599af709f"} Sep 10 15:18:19.383874 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.383817 2581 scope.go:117] "RemoveContainer" containerID="d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5" Sep 10 15:18:19.384294 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:18:19.384164 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5\": container with ID starting with d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5 not found: ID does not exist" containerID="d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5" Sep 10 15:18:19.384294 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.384189 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5"} err="failed to get container status \"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5\": rpc error: code = NotFound desc = could not find container \"d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5\": container with ID starting with d1e8c609305982a9c4c225ed4c43b1a1294e78077e20d7a4cb9526f584361dc5 not found: ID does not exist" Sep 10 15:18:19.384294 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.384206 2581 scope.go:117] "RemoveContainer" containerID="c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0" Sep 10 15:18:19.393198 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.393176 2581 scope.go:117] "RemoveContainer" containerID="c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0" Sep 10 15:18:19.393520 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:18:19.393497 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0\": container with ID starting with c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0 not found: ID does not exist" containerID="c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0" Sep 10 15:18:19.393590 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.393527 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0"} err="failed to get container status \"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0\": rpc error: code = NotFound desc = could not find container \"c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0\": container with ID starting with c867414d2692f3b8d1295f607713303f7860cab409c835fe04f4cd4f147907a0 not found: ID does not exist" Sep 10 15:18:19.396082 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.396057 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:18:19.399553 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.399529 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d5a2df-9764d67cc-2xm2s"] Sep 10 15:18:19.408670 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.408637 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:18:19.412925 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:19.412902 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d5a2df-85b6db4f94-l6vb9"] Sep 10 15:18:20.229552 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:20.229521 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033b206f-18ee-4703-bc44-266e9ce196d9" path="/var/lib/kubelet/pods/033b206f-18ee-4703-bc44-266e9ce196d9/volumes" Sep 10 15:18:20.229971 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:20.229893 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" path="/var/lib/kubelet/pods/2b5b0cca-6c3c-49f5-a9c9-510470ba29aa/volumes" Sep 10 15:18:20.230137 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:20.230124 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78ecda3-7e57-4265-92ff-1f60ac824f0b" path="/var/lib/kubelet/pods/a78ecda3-7e57-4265-92ff-1f60ac824f0b/volumes" Sep 10 15:18:24.892568 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:24.892533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:24.994504 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:24.994467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:25.155551 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:25.155466 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:18:26.686952 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:26.686921 2581 scope.go:117] "RemoveContainer" containerID="3dc47ff6019337c63b57aaaee945c76db6a1dfc0c803679d503ca4160b9dab79" Sep 10 15:18:27.670412 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:27.670375 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 15:18:27.670770 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:27.670723 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" podUID="d15b707f-ce62-43ab-8d86-d879f33b9422" containerName="manager" containerID="cri-o://0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff" gracePeriod=30 Sep 10 15:18:27.924769 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:27.924696 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 15:18:28.023035 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.022998 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") pod \"d15b707f-ce62-43ab-8d86-d879f33b9422\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " Sep 10 15:18:28.023240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.023055 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") pod \"d15b707f-ce62-43ab-8d86-d879f33b9422\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " Sep 10 15:18:28.023240 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.023121 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f\") pod \"d15b707f-ce62-43ab-8d86-d879f33b9422\" (UID: \"d15b707f-ce62-43ab-8d86-d879f33b9422\") " Sep 10 15:18:28.025389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.025349 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d15b707f-ce62-43ab-8d86-d879f33b9422" (UID: "d15b707f-ce62-43ab-8d86-d879f33b9422"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:18:28.025389 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.025371 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f" (OuterVolumeSpecName: "kube-api-access-9tf4f") pod "d15b707f-ce62-43ab-8d86-d879f33b9422" (UID: "d15b707f-ce62-43ab-8d86-d879f33b9422"). InnerVolumeSpecName "kube-api-access-9tf4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:28.025583 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.025414 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d15b707f-ce62-43ab-8d86-d879f33b9422" (UID: "d15b707f-ce62-43ab-8d86-d879f33b9422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:18:28.124727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.124687 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-metrics-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:28.124727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.124717 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d15b707f-ce62-43ab-8d86-d879f33b9422-webhook-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:28.124727 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.124725 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/d15b707f-ce62-43ab-8d86-d879f33b9422-kube-api-access-9tf4f\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:18:28.428278 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.428246 2581 generic.go:356] "Generic (PLEG): container finished" podID="d15b707f-ce62-43ab-8d86-d879f33b9422" containerID="0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff" exitCode=0 Sep 10 15:18:28.428465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.428296 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" Sep 10 15:18:28.428465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.428322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" event={"ID":"d15b707f-ce62-43ab-8d86-d879f33b9422","Type":"ContainerDied","Data":"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff"} Sep 10 15:18:28.428465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.428374 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-7lmtq" event={"ID":"d15b707f-ce62-43ab-8d86-d879f33b9422","Type":"ContainerDied","Data":"87cb4469ae19d1b2619efc73b2c50139bc82d5ddf39cbc940f7a92a8bceb6f07"} Sep 10 15:18:28.428465 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.428396 2581 scope.go:117] "RemoveContainer" containerID="0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff" Sep 10 15:18:28.437458 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.437443 2581 scope.go:117] "RemoveContainer" containerID="0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff" Sep 10 15:18:28.437656 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:18:28.437638 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff\": container with ID starting with 0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff not found: ID does not exist" containerID="0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff" Sep 10 15:18:28.437706 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.437682 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff"} err="failed to get container status \"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff\": rpc error: code = NotFound desc = could not find container \"0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff\": container with ID starting with 0db7c71207477b02ca1dabd527f2458fc2d830948ac01e862377035bcb96ccff not found: ID does not exist" Sep 10 15:18:28.446435 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.446412 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 15:18:28.451198 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:28.451170 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-7lmtq"] Sep 10 15:18:30.230057 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:18:30.230025 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15b707f-ce62-43ab-8d86-d879f33b9422" path="/var/lib/kubelet/pods/d15b707f-ce62-43ab-8d86-d879f33b9422/volumes" Sep 10 15:19:11.995126 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:11.995092 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:19:12.095417 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:12.095369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:19:26.737872 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:26.737835 2581 scope.go:117] "RemoveContainer" containerID="680799365e4aa7c9daeffd504ff971cba6c854aa5228cd5dae33d04b5d29773c" Sep 10 15:19:26.747908 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:26.747889 2581 scope.go:117] "RemoveContainer" containerID="ab696d4ccb4b312b677fb8bfdb7f808140213f8d50959bd16018d0319a2ae353" Sep 10 15:19:26.756139 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:26.756119 2581 scope.go:117] "RemoveContainer" containerID="2c106d6e5b819e00b40ae76fa977b58c232a4ae63393768989fc23fc4244ecd2" Sep 10 15:19:29.246203 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246170 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246515 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a4273e-534e-4351-a016-b1199481fbf9" containerName="revoke-keys" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246526 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a4273e-534e-4351-a016-b1199481fbf9" containerName="revoke-keys" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246540 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15b707f-ce62-43ab-8d86-d879f33b9422" containerName="manager" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246546 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15b707f-ce62-43ab-8d86-d879f33b9422" containerName="manager" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246554 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" containerName="payload-pre-processing" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246559 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" containerName="payload-pre-processing" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246566 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" containerName="payload-pre-processing" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246571 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" containerName="payload-pre-processing" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246584 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033b206f-18ee-4703-bc44-266e9ce196d9" containerName="payload-processing" Sep 10 15:19:29.246585 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246590 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="033b206f-18ee-4703-bc44-266e9ce196d9" containerName="payload-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246597 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" containerName="payload-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246602 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" containerName="payload-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246658 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8953ab1-88fc-4a78-8d63-5d4ef64abc87" containerName="payload-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246665 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5b0cca-6c3c-49f5-a9c9-510470ba29aa" containerName="payload-pre-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246674 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15b707f-ce62-43ab-8d86-d879f33b9422" containerName="manager" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246680 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="033b206f-18ee-4703-bc44-266e9ce196d9" containerName="payload-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246685 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482ff74-63e4-401d-80c5-4b4c1ea224a1" containerName="payload-pre-processing" Sep 10 15:19:29.246934 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.246691 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a4273e-534e-4351-a016-b1199481fbf9" containerName="revoke-keys" Sep 10 15:19:29.249703 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.249677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.253135 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.253105 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 15:19:29.253135 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.253120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 15:19:29.253328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.253144 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nmwpz\"" Sep 10 15:19:29.253328 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.253172 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 15:19:29.260403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.260382 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:29.372430 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.372381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djdg\" (UniqueName: \"kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.372430 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.372434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.372652 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.372527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.473153 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.473113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9djdg\" (UniqueName: \"kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.473153 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.473160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.473418 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.473212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.475861 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.475835 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.475973 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.475898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.481903 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.481882 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djdg\" (UniqueName: \"kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg\") pod \"maas-controller-697fdc88d8-phbsc\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.559978 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.559898 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:29.834941 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.834830 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:29.837231 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:19:29.837199 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93630a77_3049_4766_b348_964817ad94d3.slice/crio-b497d4092bbf487c08a82bf5155863fe6935d7f1a1852e020e2f794d6d1800f5 WatchSource:0}: Error finding container b497d4092bbf487c08a82bf5155863fe6935d7f1a1852e020e2f794d6d1800f5: Status 404 returned error can't find the container with id b497d4092bbf487c08a82bf5155863fe6935d7f1a1852e020e2f794d6d1800f5 Sep 10 15:19:29.838572 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:29.838554 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:19:30.761810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:30.761778 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-phbsc" event={"ID":"93630a77-3049-4766-b348-964817ad94d3","Type":"ContainerStarted","Data":"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22"} Sep 10 15:19:30.762197 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:30.761827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-phbsc" event={"ID":"93630a77-3049-4766-b348-964817ad94d3","Type":"ContainerStarted","Data":"b497d4092bbf487c08a82bf5155863fe6935d7f1a1852e020e2f794d6d1800f5"} Sep 10 15:19:30.762197 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:30.761862 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:30.777561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:30.777505 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-697fdc88d8-phbsc" podStartSLOduration=1.242676261 podStartE2EDuration="1.777490778s" totalImagesPullingTime="534.814517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:19:30.777285663 +0000 UTC m=+1925.071587792" watchObservedRunningTime="2026-09-10 15:19:30.777490778 +0000 UTC m=+1925.071792907" Sep 10 15:19:41.769847 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:41.769815 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:49.403626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:49.403593 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:19:50.957134 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:50.957102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:19:51.055939 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.055900 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:51.056156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.056133 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-697fdc88d8-phbsc" podUID="93630a77-3049-4766-b348-964817ad94d3" containerName="manager" containerID="cri-o://abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22" gracePeriod=30 Sep 10 15:19:51.308138 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.308104 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:51.469252 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.469209 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert\") pod \"93630a77-3049-4766-b348-964817ad94d3\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " Sep 10 15:19:51.469252 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.469255 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9djdg\" (UniqueName: \"kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg\") pod \"93630a77-3049-4766-b348-964817ad94d3\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " Sep 10 15:19:51.469478 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.469299 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs\") pod \"93630a77-3049-4766-b348-964817ad94d3\" (UID: \"93630a77-3049-4766-b348-964817ad94d3\") " Sep 10 15:19:51.471676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.471597 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "93630a77-3049-4766-b348-964817ad94d3" (UID: "93630a77-3049-4766-b348-964817ad94d3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:19:51.471676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.471610 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93630a77-3049-4766-b348-964817ad94d3" (UID: "93630a77-3049-4766-b348-964817ad94d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:19:51.471676 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.471638 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg" (OuterVolumeSpecName: "kube-api-access-9djdg") pod "93630a77-3049-4766-b348-964817ad94d3" (UID: "93630a77-3049-4766-b348-964817ad94d3"). InnerVolumeSpecName "kube-api-access-9djdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:19:51.570179 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.570151 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-webhook-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:19:51.570179 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.570176 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9djdg\" (UniqueName: \"kubernetes.io/projected/93630a77-3049-4766-b348-964817ad94d3-kube-api-access-9djdg\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:19:51.570179 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.570185 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93630a77-3049-4766-b348-964817ad94d3-metrics-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:19:51.874406 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.874372 2581 generic.go:356] "Generic (PLEG): container finished" podID="93630a77-3049-4766-b348-964817ad94d3" containerID="abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22" exitCode=0 Sep 10 15:19:51.874582 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.874416 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-phbsc" Sep 10 15:19:51.874582 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.874446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-phbsc" event={"ID":"93630a77-3049-4766-b348-964817ad94d3","Type":"ContainerDied","Data":"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22"} Sep 10 15:19:51.874582 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.874485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-phbsc" event={"ID":"93630a77-3049-4766-b348-964817ad94d3","Type":"ContainerDied","Data":"b497d4092bbf487c08a82bf5155863fe6935d7f1a1852e020e2f794d6d1800f5"} Sep 10 15:19:51.874582 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.874507 2581 scope.go:117] "RemoveContainer" containerID="abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22" Sep 10 15:19:51.884313 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.884292 2581 scope.go:117] "RemoveContainer" containerID="abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22" Sep 10 15:19:51.884594 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:19:51.884572 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22\": container with ID starting with abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22 not found: ID does not exist" containerID="abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22" Sep 10 15:19:51.884645 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.884604 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22"} err="failed to get container status \"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22\": rpc error: code = NotFound desc = could not find container \"abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22\": container with ID starting with abc9332ae98d94bb98ac115c5b1f2e2faee124403caa7dac0e45dc3853b8fb22 not found: ID does not exist" Sep 10 15:19:51.896960 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.896934 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:51.902449 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:51.902426 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-phbsc"] Sep 10 15:19:52.231247 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.231160 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93630a77-3049-4766-b348-964817ad94d3" path="/var/lib/kubelet/pods/93630a77-3049-4766-b348-964817ad94d3/volumes" Sep 10 15:19:52.552151 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.552118 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:19:52.656731 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.656694 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-697fdc88d8-6hckw"] Sep 10 15:19:52.657122 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.657104 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93630a77-3049-4766-b348-964817ad94d3" containerName="manager" Sep 10 15:19:52.657122 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.657124 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93630a77-3049-4766-b348-964817ad94d3" containerName="manager" Sep 10 15:19:52.657278 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.657253 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93630a77-3049-4766-b348-964817ad94d3" containerName="manager" Sep 10 15:19:52.661621 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.661600 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.664978 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.664957 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 15:19:52.665089 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.664980 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 15:19:52.665089 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.664961 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nmwpz\"" Sep 10 15:19:52.665089 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.664960 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 15:19:52.668624 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.668601 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-6hckw"] Sep 10 15:19:52.679065 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.679026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-webhook-cert\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.679201 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.679088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbjk\" (UniqueName: \"kubernetes.io/projected/eaedbffd-b2e9-4b84-a807-7f2b4138f668-kube-api-access-5dbjk\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.679304 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.679206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-metrics-certs\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.779991 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.779945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-metrics-certs\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.780211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.780007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-webhook-cert\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.780211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.780039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbjk\" (UniqueName: \"kubernetes.io/projected/eaedbffd-b2e9-4b84-a807-7f2b4138f668-kube-api-access-5dbjk\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.782606 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.782571 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-metrics-certs\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.782723 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.782690 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eaedbffd-b2e9-4b84-a807-7f2b4138f668-webhook-cert\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.789451 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.789416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbjk\" (UniqueName: \"kubernetes.io/projected/eaedbffd-b2e9-4b84-a807-7f2b4138f668-kube-api-access-5dbjk\") pod \"maas-controller-697fdc88d8-6hckw\" (UID: \"eaedbffd-b2e9-4b84-a807-7f2b4138f668\") " pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:52.973313 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:52.973224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:53.246984 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:53.246949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-697fdc88d8-6hckw"] Sep 10 15:19:53.249173 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:19:53.249142 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaedbffd_b2e9_4b84_a807_7f2b4138f668.slice/crio-4a18a1805e9b94124b03e08018ef6b8672438b379bca723460d8737dd0ebabc2 WatchSource:0}: Error finding container 4a18a1805e9b94124b03e08018ef6b8672438b379bca723460d8737dd0ebabc2: Status 404 returned error can't find the container with id 4a18a1805e9b94124b03e08018ef6b8672438b379bca723460d8737dd0ebabc2 Sep 10 15:19:53.888112 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:53.888074 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-6hckw" event={"ID":"eaedbffd-b2e9-4b84-a807-7f2b4138f668","Type":"ContainerStarted","Data":"0239a70a350d097cf3d6edf757455ae5ee960abf8102f816fe0bc914ca4a8378"} Sep 10 15:19:53.888318 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:53.888136 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-697fdc88d8-6hckw" event={"ID":"eaedbffd-b2e9-4b84-a807-7f2b4138f668","Type":"ContainerStarted","Data":"4a18a1805e9b94124b03e08018ef6b8672438b379bca723460d8737dd0ebabc2"} Sep 10 15:19:53.888318 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:53.888172 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:19:53.907361 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:19:53.907305 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-697fdc88d8-6hckw" podStartSLOduration=1.477157262 podStartE2EDuration="1.907288447s" totalImagesPullingTime="430.131185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:19:53.904827813 +0000 UTC m=+1948.199129942" watchObservedRunningTime="2026-09-10 15:19:53.907288447 +0000 UTC m=+1948.201590577" Sep 10 15:20:04.895542 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:20:04.895508 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-697fdc88d8-6hckw" Sep 10 15:20:16.577587 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:20:16.577546 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:20:29.101408 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:20:29.101369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:21:15.154812 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.154773 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 15:21:15.155410 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.155021 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" podUID="6cea6fd3-e290-460b-9010-cab131fdd0bb" containerName="manager" containerID="cri-o://e4aaf3ad9bb50c6c0ad3ce4adfb4b746cb1006d663c8ff9443c0c037027d5d44" gracePeriod=10 Sep 10 15:21:15.332679 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.332642 2581 generic.go:356] "Generic (PLEG): container finished" podID="6cea6fd3-e290-460b-9010-cab131fdd0bb" containerID="e4aaf3ad9bb50c6c0ad3ce4adfb4b746cb1006d663c8ff9443c0c037027d5d44" exitCode=0 Sep 10 15:21:15.332838 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.332694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" event={"ID":"6cea6fd3-e290-460b-9010-cab131fdd0bb","Type":"ContainerDied","Data":"e4aaf3ad9bb50c6c0ad3ce4adfb4b746cb1006d663c8ff9443c0c037027d5d44"} Sep 10 15:21:15.415241 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.415171 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 15:21:15.471538 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.471504 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume\") pod \"6cea6fd3-e290-460b-9010-cab131fdd0bb\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " Sep 10 15:21:15.471746 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.471597 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwkh8\" (UniqueName: \"kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8\") pod \"6cea6fd3-e290-460b-9010-cab131fdd0bb\" (UID: \"6cea6fd3-e290-460b-9010-cab131fdd0bb\") " Sep 10 15:21:15.471986 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.471958 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6cea6fd3-e290-460b-9010-cab131fdd0bb" (UID: "6cea6fd3-e290-460b-9010-cab131fdd0bb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:21:15.473995 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.473961 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8" (OuterVolumeSpecName: "kube-api-access-fwkh8") pod "6cea6fd3-e290-460b-9010-cab131fdd0bb" (UID: "6cea6fd3-e290-460b-9010-cab131fdd0bb"). InnerVolumeSpecName "kube-api-access-fwkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:21:15.572913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.572869 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkh8\" (UniqueName: \"kubernetes.io/projected/6cea6fd3-e290-460b-9010-cab131fdd0bb-kube-api-access-fwkh8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:21:15.572913 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:15.572907 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd3-e290-460b-9010-cab131fdd0bb-extensions-socket-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:21:16.339156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:16.339118 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" event={"ID":"6cea6fd3-e290-460b-9010-cab131fdd0bb","Type":"ContainerDied","Data":"c59cf0f2b6dea014d8c4e6fc229dc979c3ea3598ef590666a127909326ce572e"} Sep 10 15:21:16.339619 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:16.339184 2581 scope.go:117] "RemoveContainer" containerID="e4aaf3ad9bb50c6c0ad3ce4adfb4b746cb1006d663c8ff9443c0c037027d5d44" Sep 10 15:21:16.339619 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:16.339141 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h" Sep 10 15:21:16.360142 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:16.360107 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 15:21:16.363419 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:16.363390 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-g779h"] Sep 10 15:21:18.232428 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:18.232390 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cea6fd3-e290-460b-9010-cab131fdd0bb" path="/var/lib/kubelet/pods/6cea6fd3-e290-460b-9010-cab131fdd0bb/volumes" Sep 10 15:21:26.798274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:26.798233 2581 scope.go:117] "RemoveContainer" containerID="2e5bbeb05866ea252ec0e25ffe348388b034e9232b39feb15223120723231bff" Sep 10 15:21:26.807048 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:26.807024 2581 scope.go:117] "RemoveContainer" containerID="731cda20501bdd4911ec87b3ced38525a5989974ff55f1a9aa1e9d207f601552" Sep 10 15:21:26.815260 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:21:26.815237 2581 scope.go:117] "RemoveContainer" containerID="c308bbb510f1f180dbafe5ae3332dc4b3ba0687dbabf8f2e7043607a899d21e8" Sep 10 15:22:15.859315 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.859275 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp"] Sep 10 15:22:15.859856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.859658 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cea6fd3-e290-460b-9010-cab131fdd0bb" containerName="manager" Sep 10 15:22:15.859856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.859670 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cea6fd3-e290-460b-9010-cab131fdd0bb" containerName="manager" Sep 10 15:22:15.859856 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.859741 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cea6fd3-e290-460b-9010-cab131fdd0bb" containerName="manager" Sep 10 15:22:15.862701 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.862684 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.865487 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.865463 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hwqjk\"" Sep 10 15:22:15.874578 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.874553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp"] Sep 10 15:22:15.879271 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.879243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxlj\" (UniqueName: \"kubernetes.io/projected/573f6727-babd-4e0c-b520-155627e92677-kube-api-access-bbxlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.879396 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.879300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/573f6727-babd-4e0c-b520-155627e92677-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.979728 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.979689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxlj\" (UniqueName: \"kubernetes.io/projected/573f6727-babd-4e0c-b520-155627e92677-kube-api-access-bbxlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.979942 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.979740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/573f6727-babd-4e0c-b520-155627e92677-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.980102 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.980083 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/573f6727-babd-4e0c-b520-155627e92677-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:15.988127 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:15.988104 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxlj\" (UniqueName: \"kubernetes.io/projected/573f6727-babd-4e0c-b520-155627e92677-kube-api-access-bbxlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ljvhp\" (UID: \"573f6727-babd-4e0c-b520-155627e92677\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:16.172975 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.172859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:16.453588 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.453508 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp"] Sep 10 15:22:16.457317 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:22:16.457283 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573f6727_babd_4e0c_b520_155627e92677.slice/crio-8f02f52581e43b67b5b779854d1884a40b2d2381f55a0d45e7797545df472084 WatchSource:0}: Error finding container 8f02f52581e43b67b5b779854d1884a40b2d2381f55a0d45e7797545df472084: Status 404 returned error can't find the container with id 8f02f52581e43b67b5b779854d1884a40b2d2381f55a0d45e7797545df472084 Sep 10 15:22:16.643555 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.643516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" event={"ID":"573f6727-babd-4e0c-b520-155627e92677","Type":"ContainerStarted","Data":"24c4762e1bdf226145991f0a316608887b55ff63b9ef305d6de2146499d049d2"} Sep 10 15:22:16.643789 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.643576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" event={"ID":"573f6727-babd-4e0c-b520-155627e92677","Type":"ContainerStarted","Data":"8f02f52581e43b67b5b779854d1884a40b2d2381f55a0d45e7797545df472084"} Sep 10 15:22:16.643789 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.643650 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:22:16.666594 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:16.666543 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" podStartSLOduration=1.666527777 podStartE2EDuration="1.666527777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:16.665349861 +0000 UTC m=+2090.959651993" watchObservedRunningTime="2026-09-10 15:22:16.666527777 +0000 UTC m=+2090.960829905" Sep 10 15:22:26.345015 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:26.344979 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:22:26.350189 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:26.350163 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:22:26.846273 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:26.846241 2581 scope.go:117] "RemoveContainer" containerID="8d0347041b9e33a61aed2f62eb45f3f891ce8c6fbe8f410e751d32b5b94fcb5f" Sep 10 15:22:27.651064 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:22:27.651031 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ljvhp" Sep 10 15:23:07.507051 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:07.507017 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:23:26.869270 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:26.869236 2581 scope.go:117] "RemoveContainer" containerID="195f60150a25663f1355d1b5b0d7298bcb9c1c44611b0a56f0608aed0cc40673" Sep 10 15:23:39.854631 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.854589 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:23:39.858175 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.858149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.860857 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.860832 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-15d93b6e-kserve-self-signed-certs\"" Sep 10 15:23:39.870421 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.870393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:23:39.873212 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.873348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873277 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.873348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7dl\" (UniqueName: \"kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.873348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.873461 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.873461 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.873373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.906027 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.905990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:23:39.973776 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.973979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.973979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7dl\" (UniqueName: \"kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.973979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.973979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.973979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.973969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.974258 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.974123 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.974348 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.974325 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.974403 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.974342 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.976495 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.976461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.976607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.976594 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:39.983098 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:39.983069 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7dl\" (UniqueName: \"kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl\") pod \"e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:40.168838 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:40.168705 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:40.445068 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:40.445040 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:23:40.447856 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:23:40.447826 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd15cb84f_e1ce_4544_8dc6_c99d8c0edd91.slice/crio-061c8f8a9860b740d564767e223bee32357ad541c6f0e5dbd6230a5b2727d6a1 WatchSource:0}: Error finding container 061c8f8a9860b740d564767e223bee32357ad541c6f0e5dbd6230a5b2727d6a1: Status 404 returned error can't find the container with id 061c8f8a9860b740d564767e223bee32357ad541c6f0e5dbd6230a5b2727d6a1 Sep 10 15:23:40.891770 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:40.891723 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:23:41.094947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:41.094895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" event={"ID":"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91","Type":"ContainerStarted","Data":"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799"} Sep 10 15:23:41.094947 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:41.094953 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" event={"ID":"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91","Type":"ContainerStarted","Data":"061c8f8a9860b740d564767e223bee32357ad541c6f0e5dbd6230a5b2727d6a1"} Sep 10 15:23:41.113051 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:41.112990 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" podStartSLOduration=2.112967684 podStartE2EDuration="2.112967684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:23:41.111275304 +0000 UTC m=+2175.405577434" watchObservedRunningTime="2026-09-10 15:23:41.112967684 +0000 UTC m=+2175.407269814" Sep 10 15:23:50.169627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:50.169589 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:50.170053 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:50.169651 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:50.182679 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:50.182650 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:23:51.158619 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:23:51.158588 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:24:16.085662 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:16.085632 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:24:17.056909 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:17.056880 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:24:17.086085 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:17.086053 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:24:21.471807 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:21.471749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:24:21.573021 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:21.572985 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:24:26.889325 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:26.889287 2581 scope.go:117] "RemoveContainer" containerID="b81b45813549f6e16ae2ba757180f48a89606cd828e0f83af3c32af054788129" Sep 10 15:24:26.898066 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:26.898048 2581 scope.go:117] "RemoveContainer" containerID="672289ac9491f7b08e4e7584cf12c6635af11ca2fffcaca004381ba3c77ef795" Sep 10 15:24:32.078001 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.077934 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" podUID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" containerName="main" containerID="cri-o://1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799" gracePeriod=45 Sep 10 15:24:32.333616 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.333548 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:24:32.335630 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.335729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335660 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.335729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335686 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl7dl\" (UniqueName: \"kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.335729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.335914 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.335995 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335973 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home" (OuterVolumeSpecName: "home") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:24:32.336049 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.335991 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache" (OuterVolumeSpecName: "model-cache") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:24:32.336091 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.336054 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:24:32.336133 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.336098 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.338046 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.338022 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl" (OuterVolumeSpecName: "kube-api-access-vl7dl") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "kube-api-access-vl7dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:24:32.338046 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.338035 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm" (OuterVolumeSpecName: "dshm") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:24:32.351281 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.351252 2581 generic.go:356] "Generic (PLEG): container finished" podID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" containerID="1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799" exitCode=0 Sep 10 15:24:32.351447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.351296 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" Sep 10 15:24:32.351447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.351316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" event={"ID":"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91","Type":"ContainerDied","Data":"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799"} Sep 10 15:24:32.351447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.351362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt" event={"ID":"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91","Type":"ContainerDied","Data":"061c8f8a9860b740d564767e223bee32357ad541c6f0e5dbd6230a5b2727d6a1"} Sep 10 15:24:32.351447 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.351381 2581 scope.go:117] "RemoveContainer" containerID="1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799" Sep 10 15:24:32.360890 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.360869 2581 scope.go:117] "RemoveContainer" containerID="1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799" Sep 10 15:24:32.361231 ip-10-0-130-182 kubenswrapper[2581]: E0910 15:24:32.361205 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799\": container with ID starting with 1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799 not found: ID does not exist" containerID="1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799" Sep 10 15:24:32.361296 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.361242 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799"} err="failed to get container status \"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799\": rpc error: code = NotFound desc = could not find container \"1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799\": container with ID starting with 1e5541ab0d6cb41df0acc48e4edb1444b59f6634499b5117a52d06e4b30ef799 not found: ID does not exist" Sep 10 15:24:32.436373 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.436329 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs\") pod \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\" (UID: \"d15cb84f-e1ce-4544-8dc6-c99d8c0edd91\") " Sep 10 15:24:32.436616 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.436596 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.436660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.436624 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.436660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.436636 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.436660 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.436651 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7dl\" (UniqueName: \"kubernetes.io/projected/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-kube-api-access-vl7dl\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.438663 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.438642 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" (UID: "d15cb84f-e1ce-4544-8dc6-c99d8c0edd91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:24:32.537094 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.537052 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Sep 10 15:24:32.672545 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.672503 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:24:32.676276 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:32.676246 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-15d93b6e-kserve-6555877d47-lr4bt"] Sep 10 15:24:34.230032 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:24:34.230004 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" path="/var/lib/kubelet/pods/d15cb84f-e1ce-4544-8dc6-c99d8c0edd91/volumes" Sep 10 15:25:09.989645 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:09.989603 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559f9485b-m8q7w_4bfa5a1e-afc4-4161-9113-6b367d546315/authorino/0.log" Sep 10 15:25:15.118000 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:15.117973 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v4fhf_0664a5c7-233f-4dbd-9c8b-189954ce8391/manager/1.log" Sep 10 15:25:15.235717 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:15.235682 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hbvz5_6c13eac1-9db5-48b3-8ed7-1425b0949669/manager/0.log" Sep 10 15:25:15.352690 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:15.352654 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-697fdc88d8-6hckw_eaedbffd-b2e9-4b84-a807-7f2b4138f668/manager/0.log" Sep 10 15:25:15.731999 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:15.731967 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4b5g9_720bcbe0-35b1-4f85-bdbb-44e3fa785d97/copy-manifests/0.log" Sep 10 15:25:15.744707 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:15.744677 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4b5g9_720bcbe0-35b1-4f85-bdbb-44e3fa785d97/manager/0.log" Sep 10 15:25:16.559237 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:16.559211 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817540-s8gpl_c45de12e-2b9c-4375-98ed-cc8438c2f36d/cleanup/0.log" Sep 10 15:25:16.766150 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:16.766119 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nxznl_af61b48a-d7b0-4b33-bdce-f6b285c8c819/postgres/0.log" Sep 10 15:25:17.500488 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:17.500453 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559f9485b-m8q7w_4bfa5a1e-afc4-4161-9113-6b367d546315/authorino/0.log" Sep 10 15:25:17.615471 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:17.615443 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rhn68_4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6/manager/0.log" Sep 10 15:25:17.721649 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:17.721615 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w5m4m_bf78bd35-1d12-42de-8357-bdc308efbe69/manager/0.log" Sep 10 15:25:17.825729 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:17.825648 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2p5vg_78066a1e-4aab-4b99-ab4b-ac180cf1514c/kuadrant-console-plugin/0.log" Sep 10 15:25:17.939773 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:17.939722 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ljvhp_573f6727-babd-4e0c-b520-155627e92677/manager/0.log" Sep 10 15:25:18.180561 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:18.180489 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-9lvc8_e73d8582-ee86-4538-adca-1967a290050d/manager/1.log" Sep 10 15:25:18.747955 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:18.747922 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-7djdw_060b937a-f5c4-4360-82ab-2842f5691632/kube-auth-proxy/0.log" Sep 10 15:25:19.089810 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:19.089716 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fhcsd_4dbff12e-19a2-4b39-a665-74b03e43df4b/payload-pre-processing/0.log" Sep 10 15:25:19.201045 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:19.201016 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:25:19.653150 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:19.653115 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vwh86_98543b7a-af13-4d3d-9a06-c9820094a9c3/main/0.log" Sep 10 15:25:19.986274 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:19.986192 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2cqpz_13929071-919f-4bb6-9498-d3bf7c8764a3/main/0.log" Sep 10 15:25:25.672192 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.672161 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h5vch/must-gather-hk7kf"] Sep 10 15:25:25.672617 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.672538 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" containerName="main" Sep 10 15:25:25.672617 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.672550 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" containerName="main" Sep 10 15:25:25.672617 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.672613 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15cb84f-e1ce-4544-8dc6-c99d8c0edd91" containerName="main" Sep 10 15:25:25.676211 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.676192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.678702 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.678670 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"kube-root-ca.crt\"" Sep 10 15:25:25.678839 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.678710 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h5vch\"/\"default-dockercfg-x2924\"" Sep 10 15:25:25.679442 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.679423 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"openshift-service-ca.crt\"" Sep 10 15:25:25.685176 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.685139 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:25:25.685297 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.685286 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/must-gather-hk7kf"] Sep 10 15:25:25.794364 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.794335 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:25:25.794572 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.794548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4sv\" (UniqueName: \"kubernetes.io/projected/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-kube-api-access-zf4sv\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.794627 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.794593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-must-gather-output\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.896063 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.896028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4sv\" (UniqueName: \"kubernetes.io/projected/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-kube-api-access-zf4sv\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.896237 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.896090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-must-gather-output\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.896475 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.896455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-must-gather-output\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.906922 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.906889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4sv\" (UniqueName: \"kubernetes.io/projected/37412bdc-66b7-4ef8-8741-880e2fe5ab8d-kube-api-access-zf4sv\") pod \"must-gather-hk7kf\" (UID: \"37412bdc-66b7-4ef8-8741-880e2fe5ab8d\") " pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:25.986987 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:25.986959 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/must-gather-hk7kf" Sep 10 15:25:26.261584 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:26.261501 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/must-gather-hk7kf"] Sep 10 15:25:26.264967 ip-10-0-130-182 kubenswrapper[2581]: W0910 15:25:26.264936 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37412bdc_66b7_4ef8_8741_880e2fe5ab8d.slice/crio-48855e68f47bd9bc6fb9b6cdb8d77d29be2700af0cc0e4d558987406f9d1f8dd WatchSource:0}: Error finding container 48855e68f47bd9bc6fb9b6cdb8d77d29be2700af0cc0e4d558987406f9d1f8dd: Status 404 returned error can't find the container with id 48855e68f47bd9bc6fb9b6cdb8d77d29be2700af0cc0e4d558987406f9d1f8dd Sep 10 15:25:26.266893 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:26.266874 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:25:26.624440 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:26.624352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/must-gather-hk7kf" event={"ID":"37412bdc-66b7-4ef8-8741-880e2fe5ab8d","Type":"ContainerStarted","Data":"48855e68f47bd9bc6fb9b6cdb8d77d29be2700af0cc0e4d558987406f9d1f8dd"} Sep 10 15:25:27.635533 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:27.635489 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/must-gather-hk7kf" event={"ID":"37412bdc-66b7-4ef8-8741-880e2fe5ab8d","Type":"ContainerStarted","Data":"3d35615db25a4be6295aaf48061d26fb154bf1762cb4a991a983016989c9ce23"} Sep 10 15:25:27.635969 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:27.635551 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/must-gather-hk7kf" event={"ID":"37412bdc-66b7-4ef8-8741-880e2fe5ab8d","Type":"ContainerStarted","Data":"d56d3d1aabfedaeac9821f4e0d5d2c3871b98a15fb8a36dfdfb7783cf4bb998b"} Sep 10 15:25:27.651140 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:27.651072 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h5vch/must-gather-hk7kf" podStartSLOduration=1.697661368 podStartE2EDuration="2.651050097s" totalImagesPullingTime="953.388729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:25:27.650154505 +0000 UTC m=+2281.944456646" watchObservedRunningTime="2026-09-10 15:25:27.651050097 +0000 UTC m=+2281.945352227" Sep 10 15:25:28.833226 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:28.833190 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nhsnn_d109b72e-e301-4f56-9c25-296228e6257c/global-pull-secret-syncer/0.log" Sep 10 15:25:28.933081 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:28.933045 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" Sep 10 15:25:29.042206 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:29.042168 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-182.ec2.internal_d36b70f3f303d4cb4f8a7bad8d41d64d/haproxy/0.log" Sep 10 15:25:33.589135 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.589102 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559f9485b-m8q7w_4bfa5a1e-afc4-4161-9113-6b367d546315/authorino/0.log" Sep 10 15:25:33.628040 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.628001 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rhn68_4d1c3d32-4d01-4d9f-b97a-dfaa5c1cf9b6/manager/0.log" Sep 10 15:25:33.654427 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.654390 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w5m4m_bf78bd35-1d12-42de-8357-bdc308efbe69/manager/0.log" Sep 10 15:25:33.693272 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.693223 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-2p5vg_78066a1e-4aab-4b99-ab4b-ac180cf1514c/kuadrant-console-plugin/0.log" Sep 10 15:25:33.736614 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.736576 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ljvhp_573f6727-babd-4e0c-b520-155627e92677/manager/0.log" Sep 10 15:25:33.788742 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.788715 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-9lvc8_e73d8582-ee86-4538-adca-1967a290050d/manager/0.log" Sep 10 15:25:33.847331 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:33.845315 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-9lvc8_e73d8582-ee86-4538-adca-1967a290050d/manager/1.log" Sep 10 15:25:35.864076 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:35.864031 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc59w_e21bca5a-adea-4fc4-865d-aa2d5b68d18a/node-exporter/0.log" Sep 10 15:25:35.911654 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:35.911593 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc59w_e21bca5a-adea-4fc4-865d-aa2d5b68d18a/kube-rbac-proxy/0.log" Sep 10 15:25:35.966319 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:35.966277 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zc59w_e21bca5a-adea-4fc4-865d-aa2d5b68d18a/init-textfile/0.log" Sep 10 15:25:38.246331 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.246292 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-txkpc_e2a41a6f-a85e-4b46-b718-4a0ec8974874/networking-console-plugin/0.log" Sep 10 15:25:38.844749 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.844718 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/1.log" Sep 10 15:25:38.851689 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.851656 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xpx2r_7c353f70-b9cf-4fd2-9af1-d5bb5f0b9e84/console-operator/2.log" Sep 10 15:25:38.969939 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.969903 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj"] Sep 10 15:25:38.975633 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.975596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:38.984955 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:38.984925 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj"] Sep 10 15:25:39.034276 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.034238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-podres\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.034480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.034309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rssv6\" (UniqueName: \"kubernetes.io/projected/d11184f6-3131-460f-86dc-67f323fd2cc1-kube-api-access-rssv6\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.034480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.034346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-lib-modules\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.034480 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.034384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-proc\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.034672 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.034541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-sys\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.135750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-lib-modules\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.135750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-proc\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.135750 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135751 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-sys\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-podres\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rssv6\" (UniqueName: \"kubernetes.io/projected/d11184f6-3131-460f-86dc-67f323fd2cc1-kube-api-access-rssv6\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135885 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-proc\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135888 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-lib-modules\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135906 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-sys\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.136060 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.135989 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d11184f6-3131-460f-86dc-67f323fd2cc1-podres\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.144979 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.144950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rssv6\" (UniqueName: \"kubernetes.io/projected/d11184f6-3131-460f-86dc-67f323fd2cc1-kube-api-access-rssv6\") pod \"perf-node-gather-daemonset-9pnhj\" (UID: \"d11184f6-3131-460f-86dc-67f323fd2cc1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.298816 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.298774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:39.605614 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.605570 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj"] Sep 10 15:25:39.725749 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:39.725705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" event={"ID":"d11184f6-3131-460f-86dc-67f323fd2cc1","Type":"ContainerStarted","Data":"0baf514540c1c5440326a6825a353c5ea6ad0dc4c53271eef9deb4d12b25f14d"} Sep 10 15:25:40.168125 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:40.168067 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-2tvmp_d5fd22a3-1b7b-412c-8ae4-bb91cd16cf32/volume-data-source-validator/0.log" Sep 10 15:25:40.732446 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:40.732404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" event={"ID":"d11184f6-3131-460f-86dc-67f323fd2cc1","Type":"ContainerStarted","Data":"05b15d9e3c416679138b4c9c172c8f19601e2410ae6ddb85106f4e1441d73a74"} Sep 10 15:25:40.732954 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:40.732704 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:40.754002 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:40.753933 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" podStartSLOduration=2.753908387 podStartE2EDuration="2.753908387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:25:40.750089174 +0000 UTC m=+2295.044391294" watchObservedRunningTime="2026-09-10 15:25:40.753908387 +0000 UTC m=+2295.048210518" Sep 10 15:25:41.304797 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:41.304769 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmd2s_57af0fe3-cfc0-4fad-a725-fae7deb33ab0/dns/0.log" Sep 10 15:25:41.327797 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:41.327748 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmd2s_57af0fe3-cfc0-4fad-a725-fae7deb33ab0/kube-rbac-proxy/0.log" Sep 10 15:25:41.427957 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:41.427925 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9gr4p_33eedfe2-81ba-4f3a-ba9c-5206a372afd8/dns-node-resolver/0.log" Sep 10 15:25:42.052743 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:42.052711 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dp6lf_5d602dc7-5a71-455b-bc86-7f096b2cb9c8/node-ca/0.log" Sep 10 15:25:43.472450 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:43.472421 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-7djdw_060b937a-f5c4-4360-82ab-2842f5691632/kube-auth-proxy/0.log" Sep 10 15:25:43.581940 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:43.581906 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fhcsd_4dbff12e-19a2-4b39-a665-74b03e43df4b/payload-pre-processing/0.log" Sep 10 15:25:43.620245 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:43.620214 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:25:44.129709 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:44.129673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2xbnw_2582a9f4-efac-41a8-abb1-3ac3f0c04392/konnectivity-agent/0.log" limit=1024 Sep 10 15:25:44.522386 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:44.522357 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hr8bk_c6af49c5-244d-47de-bb7f-3d6cfde75f04/serve-healthcheck-canary/0.log" Sep 10 15:25:45.179469 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:45.179444 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vlqpb_e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b/kube-rbac-proxy/0.log" Sep 10 15:25:45.200907 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:45.200879 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vlqpb_e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b/exporter/0.log" Sep 10 15:25:45.223285 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:45.223257 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vlqpb_e9e0019b-c19f-4ee9-ae9b-c9743a75fd0b/extractor/0.log" Sep 10 15:25:46.752828 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:46.752797 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-9pnhj" Sep 10 15:25:47.675358 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:47.675324 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v4fhf_0664a5c7-233f-4dbd-9c8b-189954ce8391/manager/0.log" Sep 10 15:25:47.675518 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:47.675412 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v4fhf_0664a5c7-233f-4dbd-9c8b-189954ce8391/manager/1.log" Sep 10 15:25:47.908156 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:47.908074 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hbvz5_6c13eac1-9db5-48b3-8ed7-1425b0949669/manager/0.log" Sep 10 15:25:47.964924 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:47.964894 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-697fdc88d8-6hckw_eaedbffd-b2e9-4b84-a807-7f2b4138f668/manager/0.log" Sep 10 15:25:48.212924 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:48.212847 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4b5g9_720bcbe0-35b1-4f85-bdbb-44e3fa785d97/manager/0.log" Sep 10 15:25:48.236868 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:48.236823 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4b5g9_720bcbe0-35b1-4f85-bdbb-44e3fa785d97/copy-manifests/0.log" Sep 10 15:25:50.549889 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:50.549860 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-7djdw_060b937a-f5c4-4360-82ab-2842f5691632/kube-auth-proxy/0.log" Sep 10 15:25:50.644779 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:50.644648 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fhcsd_4dbff12e-19a2-4b39-a665-74b03e43df4b/payload-pre-processing/0.log" Sep 10 15:25:50.673329 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:50.673213 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:25:52.605052 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:52.605005 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-7djdw_060b937a-f5c4-4360-82ab-2842f5691632/kube-auth-proxy/0.log" Sep 10 15:25:52.688321 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:52.688290 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fhcsd_4dbff12e-19a2-4b39-a665-74b03e43df4b/payload-pre-processing/0.log" Sep 10 15:25:52.719607 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:52.719569 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nxzvd_ceda0459-fbdc-4ba2-b8be-9235714cf57d/payload-processing/0.log" Sep 10 15:25:55.252286 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:55.252249 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-jhd4p_b7d8884e-96c9-435b-8150-175a79139143/kube-storage-version-migrator-operator/1.log" Sep 10 15:25:55.253282 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:55.253259 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-jhd4p_b7d8884e-96c9-435b-8150-175a79139143/kube-storage-version-migrator-operator/0.log" Sep 10 15:25:56.433149 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.433122 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/kube-multus-additional-cni-plugins/0.log" Sep 10 15:25:56.456314 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.456276 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/egress-router-binary-copy/0.log" Sep 10 15:25:56.478085 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.478054 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/cni-plugins/0.log" Sep 10 15:25:56.500158 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.500130 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/bond-cni-plugin/0.log" Sep 10 15:25:56.523907 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.523865 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/routeoverride-cni/0.log" Sep 10 15:25:56.546626 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.546600 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/whereabouts-cni-bincopy/0.log" Sep 10 15:25:56.584851 ip-10-0-130-182 kubenswrapper[2581]: I0910 15:25:56.584820 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvfqw_7009b52c-fa14-4627-af75-b0af34939d26/whereabouts-cni/0.log"